# InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models

**Authors:** Lichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang, Tianyi Zhou  
**arXiv:** [2306.03082](https://arxiv.org/abs/2306.03082)  
**Date:** 2023-06-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.03082v1/

## Key points

- Large language models (LLMs) have improved performance across various downstream applications in natural language processing (NLP) tasks.
- Finding the best instruction for different situations can be challenging, especially for black-box LLMs where backpropagation is forbidden.
- The authors propose a novel method called InstructZero to optimize a low-dimensional soft prompt applied to an open-source LLM to generate instructions for black-box LLMs.
- InstructZero converts the soft prompt into an instruction using the open-source LLM and submits it to the black-box LLM for zero-shot evaluation.
- The resulting performance is then sent to Bayesian optimization to produce new soft prompts that improve zero-shot performance.
- InstructZero outperforms state-of-the-art auto-instruction methods across a variety of downstream tasks and has potential implications in various NLP applications where accurate instructions are crucial but difficult to obtain manually.
- Experiments show that InstructZero has the potential to break the scaling law of LLMs as a 10x smaller open-source model can be used to optimize an instruction with superior performance compared to a much larger LLM.
- A case study demonstrates how INSTRUCTZERO generates instructions with high execution accuracy compared to two baselines: manual input with exemplars and input without manual prompts.
- An ablation study comparing INSTRUCTZERO's optimized instructions with those generated by Uniform and APE methods for various tasks such as formality, cause-and effect, negation, translation_en–fr, sum letters_list and larger_animal was conducted.

## Summary

Large language models (LLMs) have revolutionized natural language processing (NLP) tasks by consistently improving performance across various downstream applications. However, finding the best instruction for different situations can be challenging, especially for black-box LLMs where backpropagation is forbidden. In this paper, the authors propose a novel method called InstructZero to optimize a low-dimensional soft prompt applied to an open-source LLM to generate instructions for black-box LLMs. The proposed method converts the soft prompt into an instruction using the open-source LLM and submits it to the black-box LLM for zero-shot evaluation. The resulting performance is then sent to Bayesian optimization to produce new soft prompts that improve zero-shot performance. The authors evaluate InstructZero on different combinations of open-source LLMs and APIs, including Vicuna and ChatGPT, and demonstrate that it outperforms state-of-the-art auto-instruction methods across a variety of downstream tasks. Furthermore, experiments show that InstructZero has the potential to break the scaling law of LLMs as a 10x smaller open-source model can be used to optimize an instruction with superior performance compared to a much larger LLM. In addition, the authors present a case study demonstrating how INSTRUCTZERO generates instructions with high execution accuracy compared to two baselines: manual input with exemplars and input without manual prompts. They also conduct an ablation study comparing INSTRUCTZERO's optimized instructions with those generated by Uniform and APE methods for various tasks such as formality, cause-and effect, negation, translation_en–fr, sum letters_list and larger_animal. Overall, this paper presents a promising approach for optimizing instructions in black box LLMs using soft prompts generated by open source models. The proposed method has potential implications in various NLP applications where accurate instructions are crucial but difficult to obtain manually.