# Attack Prompt Generation for Red Teaming and Defending Large Language Models

**Authors:** Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, Xiangnan He  
**arXiv:** [2310.12505](https://arxiv.org/abs/2310.12505)  
**Date:** 2023-10-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.12505v1/

## Key points

- Authors address vulnerability of large language models (LLMs) to red teaming attacks
- Proposed integrated approach combines manual and automatic methods for generating high-quality attack prompts
- Introduction of attack framework guiding LLMs to emulate human-generated prompts through in-context learning
- Defense framework proposed to fine-tune victim LLMs iteratively to enhance resilience against attacks
- Extensive experiments validate effectiveness of attack and defense frameworks on various LLMs
- Release of attack prompt datasets named SAP in different sizes for evaluation and improvement of safety measures
- Code and dataset available on GitHub for further exploration and implementation

## Summary

In their paper titled "Attack Prompt Generation for Red Teaming and Defending Large Language Models," authors Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, and Xiangnan He address the vulnerability of large language models (LLMs) to red teaming attacks that can lead them to produce harmful content. The authors propose an integrated approach that combines manual and automatic methods to efficiently generate high-quality attack prompts in order to overcome limitations in cost and quality. Leveraging the advanced capabilities of newly emerged LLMs, they introduce an attack framework that guides LLMs to emulate human-generated prompts through in-context learning. Additionally, a defense framework is proposed to fine-tune victim LLMs by iteratively interacting with the attack framework to enhance their resilience against red teaming attacks. Extensive experiments conducted on various LLMs validate the effectiveness of the proposed attack and defense frameworks. Furthermore, the authors release a series of attack prompt datasets named SAP in different sizes to facilitate evaluation and improvement of safety measures for a broader range of LLMs. The code and dataset are available on GitHub for further exploration and implementation. This research was accepted at EMNLP 2023 (Findings) and significantly contributes to enhancing the security of large language models against potential adversarial attacks.