Running summarizing tools on a new article

Abstract: Retrieval-Augmented Generation (RAG) ranks passages by their relevance to a question and passes the top ones to the generator whether or not they contain the answer. Relevance, however, does not imply sufficiency: a passage can be on topic and still lack the fact the question asks for. We place a sufficiency gate between retriever and generator that judges whether the retrieved context is adequate and then answers, abstains, or retrieves more, and we compare four scorers built without any sufficiency-labelled training data—a simple heuristic, a single-passage cross-encoder, a set-level cross-encoder that scores the retrieved passages jointly, and the generator itself prompted as a judge—on 500 HotpotQA and 500 MuSiQue multi-hop questions with Flan-T5-Large. An answer is counted as unsupported when no retrieved passage entails it according to a natural language inference model (DeBERTa-large-MNLI) at a threshold fixed once on held-out data. Because abstaining lowers any error count that ignores unanswered questions, we report the unsupported rate among answered questions alongside coverage. Vanilla RAG is unsupported on 26.2% of HotpotQA and 30.8% of MuSiQue answers. At their calibrated operating points the set-level cross-encoder is the best gate on HotpotQA (17.1% unsupported at 57.2% coverage, with 7.0% of sufficient contexts wrongly refused) and the generator-as-judge on MuSiQue (27.0% at 71.0% coverage, none wrongly refused). Only the judge beats a random abstainer matched to its own coverage on both datasets (both p<0.01), and only the judge ranks best at every threshold. Reliability is paid for in usefulness: on HotpotQA, Exact Match rises from 32.0% to 43.9% among answered questions but falls from 32.0% to 25.0% when abstentions are scored as wrong. Both cross-encoders identify insufficient context better than the heuristic by the area under the receiver operating characteristic curve, decisively on HotpotQA (0.775 and 0.675 against 0.521) but within the noise on MuSiQue (0.738 and 0.717 against 0.658); by selective risk across all operating points the heuristic is worse than not gating at all. Replacing the retrieved context with the gold supporting passages removes about two thirds of the answer-quality failures and between a third and a half of the unsupported answers, so much of the failure, though not all, originates in retrieval rather than the generator. Repeating the evaluation with Qwen2.5-3B-Instruct shifts the absolute rates but preserves the ordering of the scorers.