# Inspecting and Editing Knowledge Representations in Language Models

**Authors:** Evan Hernandez, Belinda Z. Li, Jacob Andreas  
**arXiv:** [2304.00740](https://arxiv.org/abs/2304.00740)  
**Date:** 2023-04-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.00740v2/

## Key points

- The paper explores the representation of facts in neural language models (LMs)
- LMs encode facts based on training data and input text
- REMEDI is introduced as a method for mapping natural language statements to fact encodings within an LM's internal representation system
- REMEDI encodings serve as knowledge editors, allowing modifications to downstream generation for consistency with new facts
- REMEDI encodings can be used as probes to compare with LM representations and identify properties attributed by LMs to mentioned entities
- REMEDI offers a step towards developing general tools for inspection and control of knowledge in LMs
- Understanding and managing factual representations within LMs is important for enhancing reliability and aligning them with real-world knowledge
- The goal is to create more reliable systems that accurately represent factual information from different sources

## Summary

The paper titled "Inspecting and Editing Knowledge Representations in Language Models" by Evan Hernandez, Belinda Z. Li, and Jacob Andreas explores the representation of facts in neural language models (LMs). LMs encode facts about the world based on both training data and input text. For example, the word "banana" in an LM represents the fact that bananas are fruits, while a sentence like "I poured out the bottle" encodes the fact that the bottle became empty. The authors introduce REMEDI, a method for learning to map natural language statements to fact encodings within an LM's internal representation system. These REMEDI encodings serve as knowledge editors, allowing modifications to downstream generation to ensure consistency with new facts. Additionally, REMEDI encodings can be used as probes to compare with LM representations and identify properties attributed by LMs to mentioned entities. This enables predictions of when LMs might generate outputs conflicting with background knowledge or input text. By linking work on probing, prompting, and LM editing, REMEDI offers a step towards developing general tools for fine-grained inspection and control of knowledge in LMs. The authors emphasize the importance of understanding and managing the factual representations within LMs to enhance their reliability and align them with real-world knowledge. Through this approach they aim to create more reliable systems which can better represent factual information from different sources such as training data or user input text. Thus providing users with more accurate results which are consistent with their expectations from real-world knowledge.