# Observations on Building RAG Systems for Technical Documents

**Authors:** Sumit Soman, Sujoy Roychowdhury  
**arXiv:** [2404.00657](https://arxiv.org/abs/2404.00657)  
**Date:** 2024-03-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.00657v1/

## Key points

- Chunk length has a significant impact on retriever embeddings in RAG systems for technical documents.
- Relying solely on similarity scores to augment the generator may not always be reliable.
- Abbreviations and a large number of related paragraphs are relevant for long-form Question Answering (QA) in technical documents.
- Future work includes incorporating RAG metrics proposed by Es et al. and Chen et al., developing effective methods, and evaluation metrics for addressing follow-up questions within the RAG framework.
- Experiments focused on IEEE Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) specifications, as well as the IEEE Standard Glossary of Stationary Battery Terminology.
- Observations showed that sentence embeddings become less reliable with increasing chunk size, leading to spurious similarities that were manually validated for accuracy.
- When both query and queried document exceeded 200 words, similarity distributions exhibited a bimodal nature.

## Summary

, , , , 

The study on building Retrieval Augmented Generation (RAG) systems for technical documents revealed the significant impact of chunk length on retriever embeddings. It was also noted that relying solely on similarity scores to augment the generator may not always be reliable. The use of abbreviations and a large number of related paragraphs were found to be particularly relevant for long-form Question Answering (QA) in technical documents. As part of future work, the researchers plan to incorporate RAG metrics proposed by Es et al. and Chen et al. to inform retrieval strategies and develop effective methods and evaluation metrics for addressing follow-up questions within the RAG framework. The experiments conducted focused on IEEE Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) specifications, as well as the IEEE Standard Glossary of Stationary Battery Terminology. By examining the influence of chunk length, keyword-based search, and rank of retrieved results in the RAG pipeline, the researchers aimed to gain a better understanding of factors affecting retrieval performance in technical document QA. Observations from the study showed that sentence embeddings become less reliable with increasing chunk size, as evidenced by a Kernel Density Estimate plot displaying high similarity scores for longer sentences. The distribution of higher similarities for larger lengths suggested spurious similarities, which were manually validated for accuracy. Additionally, it was highlighted that when both query and queried document exceeded 200 words, similarity distributions exhibited a bimodal nature. Overall, this research provides valuable insights into optimizing RAG systems for technical documents by addressing key challenges such as chunk length impact on retriever embeddings and reliability issues with generator augmentation strategies based on similarity scores. Future work will focus on leveraging advanced RAG metrics and developing innovative methods to enhance question answering capabilities within technical document contexts.