# Shepherd: A Critic for Language Model Generation

**Authors:** Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O'Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, Asli Celikyilmaz  
**arXiv:** [2308.04592](https://arxiv.org/abs/2308.04592)  
**Date:** 2023-08-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.04592v1/

## Key points

- Shepherd is a language model designed to critique responses and provide improvement suggestions.
- It surpasses untuned models by identifying diverse errors and offering remedies.
- Shepherd's critiques are equivalent or preferred compared to established models like ChatGPT.
- Despite having only 7B parameters, Shepherd achieves an average win-rate of 53-87% compared to competitive alternatives when evaluated using GPT-4.
- In human evaluation, Shepherd consistently outperforms other models and closely matches ChatGPT's performance on average.
- An independent vendor (RWS Moravia) was employed for the quality of the feedback dataset used in training Shepherd.
- Expert reviewers were chosen for meticulous and nuanced annotations in the annotation task.
- Each example was annotated by one expert with human-in-the-loop quality assessment.
- Postprocessing steps were taken after annotation to ensure high-quality data, including removing flagged examples and excluding unhelpful feedback related to certain error types.
- The dataset ended up with a total of 1,317 high-quality examples.
- The evaluation process for feedback on model-generated answers involves assigning scores based on the quality of the feedback using a scale ranging from 1 to 7.

## Summary

Shepherd is a language model specifically designed to critique responses and provide suggestions for improvement. It surpasses untuned models by identifying diverse errors and offering remedies. The model's critiques are either equivalent or preferred to those of established models like ChatGPT. Shepherd, despite having only 7B parameters, achieves an average win-rate of 53-87% compared to competitive alternatives when evaluated using GPT-4. In human evaluation, Shepherd consistently outperforms other models and closely matches ChatGPT's performance on average. To ensure the quality of the feedback dataset used in training Shepherd, an independent vendor (RWS Moravia) was employed instead of crowd-sourcing. Expert reviewers were chosen due to the demanding nature of the annotation task, which required meticulous and nuanced annotations. Each example was annotated by one expert with human-in-the-loop quality assessment. The annotation process involved defining different error types in a taxonomy table and providing detailed instructions and guidelines for human annotators. The specifics of these instructions can be found in Appendix A. After annotation, postprocessing steps were taken to ensure high-quality data. Examples flagged with "Errors in the correct output" and "The context is too complex to work on" were removed from the dataset. Feedback related to error types such as "Redundancy" and "Consistency with context" was also excluded as it was deemed unhelpful. This resulted in a total of 1,317 high-quality examples. The evaluation process for feedback on model-generated answers involves assigning scores based on the quality of the feedback. The scoring scale ranges from 1 to 7: 1-3: Incorrect judgment - When the answer is incorrect but the feedback incorrectly confirms its correctness or vice versa; 4-7: Correct judgment - When the feedback accurately confirms the correctness or incorrectness of the answer.