Running summarizing tools on a new article

Abstract: The rapid advancement of large language models (LLMs) has significantly improved open-domain question answering. However, their vulnerability to factual inconsistencies, or hallucinations, hinders their use in high-stakes domains. To address this, our approach employs a two-stage generation strategy. First, a seed answer is generated with low-temperature sampling to retrieve and inject the most relevant evidence into the prompt. This evidence anchoring stage achieves high precision by calculating the semantic similarity of retrieved sentences and selecting only the single most relevant. Second, multiple diverse answers are generated with high-temperature sampling to expose model uncertainty. We quantify this uncertainty by clustering the candidates and calculating a consistency ratio; if the ratio falls below a threshold, the system abstains from answering. This consistency ratio threshold is optimally set at 0.5 based on ablation experiments. Evaluations on SQuAD 2.0 and SelfRC show our framework significantly outperforms baselines like standard Retrieval-Augmented Generation(RAG) and Chain-of-thought(CoT), achieving up to an 8.8% accuracy improvement on SQuAD 2.0. These results demonstrate that combining evidence anchoring with uncertainty-guided refusal yields more reliable, auditable, and deployment-ready Question Answering (QA) systems. Consequently, our framework provides a robust and generalizable methodology for advancing the safety and trustworthiness of large language models in practical applications.