# Auditing large language models: a three-layered approach

**Authors:** Jakob Mökander, Jonas Schuett, Hannah Rose Kirk, Luciano Floridi  
**arXiv:** [2302.08500](https://arxiv.org/abs/2302.08500)  
**Date:** 2023-02-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.08500v2/

## Key points

- Large language models (LLMs) have revolutionized AI research but also present ethical and social challenges.
- Auditing is proposed as a governance mechanism for ethical and robust design of AI systems.
- Existing auditing procedures do not adequately address the unique challenges posed by LLMs.
- A three-layered approach to auditing LLMs is outlined: governance audits, model audits, and application audits.
- These audits can effectively identify and manage ethical and social risks associated with LLMs when conducted in a structured manner.
- The proposed blueprint holds lessons for future AI systems.
- Feasibility and effectiveness may be challenged by future developments, but it can still serve as a valuable tool for analysis and evaluation of LLMs.
- The three-layered approach should be continuously revised in response to technological advancements and regulatory landscapes.
- It complements existing governance mechanisms by enhancing procedural transparency.

## Summary

Large language models (LLMs) have revolutionized artificial intelligence (AI) research but also present significant ethical and social challenges. Auditing has been proposed as a governance mechanism to ensure the ethical and robust design and deployment of AI systems. However, existing auditing procedures do not adequately address the unique challenges posed by LLMs, which possess emergent capabilities and adaptability to various tasks. In this article, a three-layered approach to auditing LLMs is outlined. This approach includes governance audits of technology providers, model audits of LLMs after pre-training but before release, and application audits of LLM-based applications. These audits can complement each other and effectively identify and manage ethical and social risks associated with LLMs when conducted in a structured and coordinated manner. It is important to recognize the limitations of auditing LLMs and remain realistic about its potential impact. The proposed blueprint for auditing LLMs holds lessons for future AI systems as well. While the feasibility and effectiveness of this approach may be challenged by future developments such as democratization of AI capabilities or personalized language models, it can still serve as a valuable tool for technology providers and policymakers to analyze and evaluate LLMs from technical, ethical, and legal perspectives. The three-layered approach should be continuously revised in response to changing technological advancements and regulatory landscapes. It is not intended to replace existing governance mechanisms but rather complement them by enhancing procedural transparency and regularity. Stakeholders can adopt, adjust, expand, or tailor this approach according to their specific needs in different contexts.