# Multilingual Sentence-Level Semantic Search using Meta-Distillation Learning

**Authors:** Meryem M'hamdi, Jonathan May, Franck Dernoncourt, Trung Bui, Seunghyun Yoon  
**arXiv:** [2309.08185](https://arxiv.org/abs/2309.08185)  
**Date:** 2023-09-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.08185v1/

## Key points

- Multilingual semantic search involves retrieving relevant content in different language combinations
- Demand for multilingual semantic search is increasing as users need to access content in multiple languages simultaneously
- Traditional machine translation approaches are being replaced by transfer learning techniques using pre-trained multilingual Transformer-based models like M-BERT and XLM-R
- M-BERT and XLM-R still have limitations, especially for ad-hoc semantic search
- The authors propose a novel approach called MAML-Align for low-resource scenarios
- MAML-Align utilizes a Teacher model (T-MAML) for transferring knowledge from monolingual to bilingual semantic search and a Student model (S-MAML) for transferring knowledge from bilingual to multilingual semantic search
- Alignment between teacher and student models is achieved through meta-distillation learning based on Model Agnostic Meta Learner (MAML)
- Empirical experiments using sentence transformers as a baseline show that the meta-distillation approach improves upon the gains provided by MAML and outperforms naive fine tuning methods
- Multilingual meta distillation learning enhances generalization even to unseen languages
- The study highlights the importance of multilingual semantic search and presents an effective approach leveraging meta distillation learning to improve performance in low resource scenarios

## Summary

Multilingual semantic search is a challenging task that involves retrieving relevant content in different language combinations. The demand for this type of search has been increasing as users across the globe need to access content in multiple languages simultaneously. Traditional approaches relying on machine translation are being replaced by transfer learning techniques using pre-trained multilingual Transformer-based models like M-BERT and XLM-R. However, these models still have limitations, especially for ad-hoc semantic search. To address these limitations, the authors propose a novel approach called MAML-Align specifically designed for low-resource scenarios. The MAML-Align framework utilizes a Teacher model (T-MAML) specialized in transferring knowledge from monolingual to bilingual semantic search and a Student model (S-MAML) specialized in transferring knowledge from bilingual to multilingual semantic search. This alignment between teacher and student models is achieved through meta-distillation learning based on Model Agnostic Meta Learner (MAML), which allows the student model to distill knowledge from the teacher model. The authors conducted empirical experiments using sentence transformers as a strong baseline. The results demonstrate that their meta-distillation approach significantly improves upon the gains provided by MAML and outperforms naive fine tuning methods. Furthermore, they found that multilingual meta distillation learning enhances generalization even to unseen languages. Overall, this study highlights the importance of multilingual semantic search and presents a novel approach that leverages meta distillation learning to enhance the performance of low resource scenarios. The results demonstrate the effectiveness of their approach in improving the gains provided by MAML and achieving better generalization even to unseen languages.