# Recovering from Privacy-Preserving Masking with Large Language Models

**Authors:** Arpita Vats, Zhe Liu, Peng Su, Debjyoti Paul, Yingyi Ma, Yutong Pang, Zeeshan Ahmed, Ozlem Kalinli  
**arXiv:** [2309.08628](https://arxiv.org/abs/2309.08628)  
**Date:** 2023-09-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.08628v1/

## Key points

- Model adaptation is crucial for handling the discrepancy between proxy training data and actual user data.
- Storing user data raises privacy and security concerns.
- Recent research explores replacing identifying information with generic markers to address privacy concerns.
- The authors propose using large language models (LLMs) to suggest substitutes for masked tokens to preserve privacy while maintaining model effectiveness.
- Multiple pre-trained and fine-tuned LLM-based approaches are evaluated on various datasets through empirical studies.
- Models trained on obfuscation corpora achieve comparable performance to models trained on original data without token masking.
- Model adaptation has potential risks to user privacy and security.
- LLMs provide a solution that effectively addresses these concerns while maintaining model performance.
- The proposed approaches are evaluated through empirical studies, showcasing their effectiveness across different datasets.
- This work contributes insights into privacy-preserving techniques for model adaptation in the field of NLP.

## Summary

Model adaptation is a crucial step in handling the discrepancy between proxy training data and actual user data. This adaptation allows for the effective training of downstream natural language processing (NLP) models using in-domain data stored on servers or local devices. However, storing user data raises privacy and security concerns as it exposes sensitive information to potential adversaries. To address this issue, recent research has explored replacing identifying information with generic markers. In this work, the authors propose leveraging large language models (LLMs) to suggest substitutes for masked tokens in order to preserve privacy while maintaining model effectiveness. They evaluate the effectiveness of multiple pre-trained and fine-tuned LLM-based approaches on various datasets through empirical studies. The experimental results demonstrate that models trained on obfuscation corpora achieve comparable performance to models trained on original data without privacy-preserving token masking. The authors highlight the importance of model adaptation and its potential risks to user privacy and security. By utilizing LLMs, they provide a solution that effectively addresses these concerns while maintaining model performance. The proposed approaches are evaluated through empirical studies, showcasing their effectiveness across different datasets. Overall, this work contributes to the field of NLP by providing insights into privacy-preserving techniques for model adaptation. The findings demonstrate that obfuscation techniques can be successfully applied without sacrificing model performance, ensuring both user privacy and effective downstream language modeling tasks.