# On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex

**Authors:** Terry Yue Zhuo, Zhuang Li, Yujin Huang, Yuan-Fang Li, Weiqing Wang, Gholamreza Haffari, Fatemeh Shiri  
**arXiv:** [2301.12868](https://arxiv.org/abs/2301.12868)  
**Date:** 2023-01-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2301.12868v1/

## Key points

- Semantic parsing is a technique for creating structured representations of natural-language questions.
- Few-shot language models trained on code outperform traditional unimodal language models in generating these representations.
- Existing fine-tuned neural semantic parsers are vulnerable to adversarial attacks on natural-language inputs.
- Adversarial training enhances the robustness of smaller semantic parsers but is not feasible for large language models due to resource constraints.
- The paper presents an empirical study on the adversarial robustness of CODEX, a large prompt-based language model.
- State-of-the-art code-language models like CODEX are susceptible to carefully crafted adversarial examples.
- The authors propose methods for improving robustness without requiring significant amounts of labeled data or heavy computational resources.
- Different sampling strategies and linguistic complexity affect CODEX's performance, with higher lexical diversity in few-shot examples leading to stronger robustness and standard accuracy.
- Future research is needed on various adversarial training strategies for prompt-based semantic parsers and investigating robustness across different semantic parsing datasets.

## Summary

Semantic parsing is a technique used to create a structured representation of the meaning of a natural-language question. Recent advancements in few-shot language models trained on code have shown better performance in generating these representations compared to traditional unimodal language models. However, existing fine-tuned neural semantic parsers are vulnerable to adversarial attacks on natural-language inputs. Adversarial training has been proven effective for enhancing the robustness of smaller semantic parsers but is not feasible for large language models due to resource constraints. This paper presents an empirical study on the adversarial robustness of a large prompt-based language model called CODEX. The results demonstrate that state-of-the-art code-language models are susceptible to carefully crafted adversarial examples. To address this challenge, the authors propose methods for improving robustness without requiring significant amounts of labeled data or heavy computational resources. The study also investigates how different sampling strategies and linguistic complexity affect CODEX's performance and concludes that higher lexical diversity in sampled few-shot examples leads to stronger robustness and standard accuracy of CODEX. Finally, the paper calls for future research on various adversarial training strategies for prompt-based semantic parsers and expanding investigations into the robustness when varying semantic parsing datasets.