The assumption to test
A higher ROUGE score can look like proof that a summary preserves meaning. It is evidence of overlap with a reference, under the chosen metric settings. ROUGE-1 counts matching word occurrences; ROUGE-2 compares adjacent word pairs; ROUGE-L uses a longest common subsequence. These comparisons can reward useful content, but they do not establish that the summary’s claims are correct.
One word changes the instruction
Consider this illustrative reference: “The valve must remain closed during maintenance.” An incorrect summary says: “The valve must remain open during maintenance.” With lowercase words, punctuation removed and no stemming, ROUGE-1 matches six of seven words. Yet the operating instruction is reversed.
A faithful paraphrase, “Keep the valve shut while servicing it,” shares only two reference words under those settings. It receives a lower overlap score while preserving the essential instruction. These examples are illustrative, not reported test results.
Use a score as a preview
A preview helps you decide what to inspect; a review checks what matters. Before comparing scores, identify the source’s actors, actions, negations, conditions, quantities and uncertainty. Then check whether each summary preserves those relationships and omits unsupported additions.
Keep the same references, preprocessing and ROUGE variant across candidates. Pair the resulting score with a claim check against the source. A high score deserves attention, not automatic trust; a low score should prompt inspection for valid paraphrasing.
Hypothetical boundary: exact identifiers
Suppose a maintenance summary must preserve an equipment identifier exactly. “Inspect valve V17” becomes “Inspect valve V71.” Almost all wording survives, but the instruction now targets different equipment. Conversely, a clear paraphrase may preserve V17 correctly.
Decision rule: require faithful claims and exact preservation of designated identifiers. Check those identifiers character by character against the source record. Reject a candidate that fails either requirement, whatever its ROUGE score. Among candidates that pass, use overlap as one diagnostic alongside completeness and clarity.