# Towards Robust Text Retrieval with Progressive Learning

**Authors:** Tong Wu, Yulei Qin, Enwei Zhang, Zihan Xu, Yuting Gao, Ke Li, Xing Sun  
**arXiv:** [2311.11691](https://arxiv.org/abs/2311.11691)  
**Date:** 2023-11-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.11691v1/

## Key points

- Authors: Tong Wu, Yulei Qin, Enwei Zhang, Zihan Xu, Yuting Gao, Ke Li, Xing Sun
- Introduces PEG model to enhance large language models (LLMs) with external knowledge sources
- Existing embedding models for text retrieval face limitations in batch sample diversity and noise levels affecting semantic correctness
- Struggle with sub-optimal convergence due to equal treatment of easy and difficult samples
- PEG model increases in-batch negative samples to 80,000 and extracts five hard negatives per query
- Incorporates progressive learning mechanism for dynamic attention modulation during training
- Trained on over 100 million data points across domains like finance, medicine, and tourism
- Covers tasks including question-answering and similarity matching
- Experimental results show PEG outperforms state-of-the-art embeddings in retrieving true positives on C-MTEB and DuReader datasets
- Publicly available at https://huggingface.co/TownsWu/PEG for further exploration and implementation in text retrieval tasks

## Summary

In their paper titled "Towards Robust Text Retrieval with Progressive Learning," authors Tong Wu, Yulei Qin, Enwei Zhang, Zihan Xu, Yuting Gao, Ke Li, and Xing Sun introduce the PEG model as a solution to enhance large language models (LLMs) with external knowledge sources for improved information handling. The existing embedding models for text retrieval face limitations in batch sample diversity and noise levels affecting semantic correctness. Additionally, they struggle with sub-optimal convergence due to equal treatment of easy and difficult samples. The PEG model addresses these challenges by increasing in-batch negative samples to 80,000 and extracting five hard negatives per query. It also incorporates a progressive learning mechanism that allows dynamic attention modulation throughout training. Trained on over 100 million data points spanning various domains such as finance, medicine, and tourism, PEG covers tasks including question-answering and similarity matching. Experimental results on C-MTEB and DuReader demonstrate that PEG outperforms state-of-the-art embeddings in retrieving true positives. This highlights the significant potential of PEG for applications in LLMs. The model is publicly available at https://huggingface.co/TownsWu/PEG for further exploration and implementation in text retrieval tasks.