Hallucination in Retrieval-Augmented Generation is a Context Construction Problem: A Constraint-Driven Cross-Modal Framework

Hallucination remains a major challenge in Retrieval-Augmented Generation (RAG), especially for domain-specific question answering over heterogeneous documents containing text, tables, figures, and OCR-derived content. This paper argues that hallucination in document-grounded RAG is strongly influenced by context construction: the quality, structure, and coherence of the retrieved evidence supplied to the generator. To address this problem, this work proposes L-DSRAG, a constraint-driven cross-modal retrieval and context construction framework that integrates structure-aware chunking, modality normalization, hybrid retrieval, graph-based expansion, table-aware grouping, cross-modal pruning, and constraint-driven context packing. Rather than passing independently retrieved top-k chunks directly to the language model, the framework treats context construction as a constrained evidence-selection problem under token and structural constraints. The system is evaluated against a traditional RAG baseline using a 100-question benchmark covering factoid, aggregation, table-based, temporal, multi-span, cross-modal, and negative/unanswerable queries. Four questions are intentionally unanswerable and are used to evaluate hallucination avoidance. Faithfulness and answer relevancy are computed over all 100 questions, while context recall and context precision are computed over the 96 answerable questions because the unanswerable questions contain no retrievable ground-truth evidence. Results show that the full proposed method improves faithfulness from 0.750 to 0.900, answer relevancy from 0.740 to 0.880, and context recall from 0.740 to 0.947. Context precision decreases from 0.737 to 0.693 over the 96 answerable questions, reflecting a tradeoff between retrieval selectivity and evidence completeness. Ablation results show that most gains come from structure-aware retrieval and table-aware context construction, while cross-modal pruning provides additional improvements by refining the final evidence context. The framework operates entirely with locally hosted components, including BAAI/bge-large-en-v1.5 embeddings and a local Qwen3-14B generator, supporting privacy-preserving and API-independent deployment. These findings show that hallucination reduction in RAG can be achieved not only through stronger generators, but through more structured and selective evidence construction before generation.