# Axiomatic Preference Modeling for Longform Question Answering

**Authors:** Corby Rosset, Guoqing Zheng, Victor Dibia, Ahmed Awadallah, Paul Bennett  
**arXiv:** [2312.02206](https://arxiv.org/abs/2312.02206)  
**Date:** 2023-12-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.02206v1/

## Key points

- Study explores the use of large language models (LLMs) like GPT-4 in longform question answering
- Focus on post-training process of Reinforcement Learning from Human Feedback (RLHF)
- RLHF involves human preferences encoded in a reward model (RM)
- Identified principles to guide RMs and developed an axiomatic framework for generating preference signals
- Trained a standalone preference model with approximately 220M parameters using axiomatic signals
- Preference Model can score both human- and LLM-generated answers on the same scale
- Contributions include outperforming GPT-4 in preference scoring, developing axiomatic framework for tailored training data pairs, and demonstrating improvement over GPT-4 with small amount of axiomatic signals
- Model released on huggingface, providing access to research findings and approach
- Paper accepted for presentation at EMNLP 2023.

## Summary

In this study titled "Axiomatic Preference Modeling for Longform Question Answering," the authors explore the use of large language models (LLMs) like GPT-4 in generating answers to longform questions. They focus on the post-training process of Reinforcement Learning from Human Feedback (RLHF), which involves human preferences encoded in a reward model (RM). To address the issue of RMs lacking direct knowledge of why certain preferences annotations were made, the authors identified principles that guide RMs to better align with human preferences and developed an axiomatic framework to generate a variety of preference signals based on these principles. They used these axiomatic signals to train a standalone preference model with approximately 220M parameters. This Preference Model is capable of scoring both human- and LLM-generated answers on the same scale. The contributions of this work include training a preference model that outperforms GPT-4 in preference scoring, developing an axiomatic framework for generating tailored training data pairs based on certain principles, and demonstrating that even a small amount of axiomatic signals can help smaller models perform better than GPT-4. The authors have released their model on huggingface, providing access to their research findings and approach. The paper has been accepted for presentation at EMNLP 2023.