# How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering

**Authors:** Zhengbao Jiang, Jun Araki, Haibo Ding, Graham Neubig  
**arXiv:** [2012.00955](https://arxiv.org/abs/2012.00955)  
**Date:** 2020-12-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2012.00955v2/

## Key points

- Authors: Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig
- Central question: How to determine when language models confidently know the answer
- Approach: Focus on calibration to align predicted probabilities with actual correctness probabilities
- Evaluation of generative models: T5, BART, GPT-2 lack calibration in question answering tasks
- Fine-tuning
- Post-hoc probability adjustments
- Modifications to predicted outputs or inputs
- Effectiveness of calibration methods demonstrated through experiments on diverse datasets
- Analysis of strengths and limitations of proposed approaches for calibrating language models
- Availability of code at https://github.com/jzbjyb/lm-calibration

## Summary

In their paper titled "How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering," authors Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig delve into the issue of language models (LM) capturing various types of knowledge but still falling short in providing accurate answers. The central question they address is how to determine when language models confidently know the answer to a specific query. They approach this inquiry through the lens of calibration, which focuses on ensuring that a probabilistic model's predicted probabilities align well with the actual probabilities of correctness. The study evaluates three prominent generative models - T5, BART, and GPT-2 - to assess whether their probabilities in question answering tasks are effectively calibrated. The findings reveal a notable lack of calibration in these models. To address this discrepancy and enhance the correlation between confidence scores and correctness likelihood, the authors explore methods such as fine-tuning, post-hoc probability adjustments, and modifications to predicted outputs or inputs. Through experiments across diverse datasets, the effectiveness of these calibration methods is demonstrated. Additionally, an analysis is conducted to identify both strengths and limitations of the proposed approaches, shedding light on potential avenues for further improvement in calibrating language models. The authors have made their code available at https://github.com/jzbjyb/lm-calibration. Overall, this research contributes valuable insights into enhancing the reliability and accuracy of language models in question answering tasks by focusing on calibration techniques.