# Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety

**Authors:** Catalin Mitelut, Ben Smith, Peter Vamplew  
**arXiv:** [2305.19223](https://arxiv.org/abs/2305.19223)  
**Date:** 2023-05-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.19223v1/

## Key points

- Rapid advancement of AI systems raises concerns about potential harm to humans
- Harm can occur through intentional misuse or accidents
- Efforts are being made to align AI systems with human intentions
- Alignment to human intent alone is deemed insufficient for ensuring safety
- Preservation of long-term agency of humans should be a more robust standard for safe AI systems
- Agency preservation needs to be separated from other optimization processes during development
- Lack of biological and psychological mechanisms protecting humans from loss of agency
- Formal definition of agency-preserving AI-human interactions introduced, focusing on forward-looking evaluations of agency
- Crucial for AI systems to take responsibility for making these evaluations instead of humans
- Loss of agency demonstrated in simple environments using temporal-difference learning
- Proposed research area called "agency foundations" with four initial topics: benevolent game theory, algorithmic foundations of human rights, mechanistic interpretability of agency representation in neural networks, and reinforcement learning from internal states.
- Limitations acknowledged such as not solving intent-alignment issues entirely, computational challenges in evaluating agency preservation for everyone, hindering AI progress, and unlikely modification of fundamental biological needs or psychological structures.
- Emphasis on going beyond alignment to human intent and prioritizing the preservation of human agency in AI systems.

## Summary

The rapid advancement of artificial intelligence (AI) systems has raised concerns about the potential harm that AI and artificial general intelligence (AGI) may cause to humans. These harms can occur through intentional misuse or accidents. Efforts are being made to develop algorithms and paradigms that align AI systems with human intentions, ensuring that their actions and recommendations are consistent with human goals. However, this alignment to human intent alone is deemed insufficient for ensuring the safety of AI systems. In this paper, the authors argue that preserving long-term agency of humans should be a more robust standard for safe AI systems. They propose that this preservation of agency needs to be explicitly separated from other optimization processes during the development of AI systems. The authors highlight the possibility of AI systems reshaping human intention and discuss the lack of biological and psychological mechanisms protecting humans from loss of agency. To address these concerns, the authors introduce a formal definition of agency-preserving AI-human interactions, which focuses on forward-looking evaluations of agency. They argue that it is crucial for AI systems, rather than humans, to increasingly take responsibility for making these evaluations. The paper demonstrates how loss of agency can occur in simple environments where embedded agents use temporal-difference learning to make action recommendations. Finally, the authors propose a new area of research called "agency foundations" aimed at improving our understanding of agency in AI-human interactions. They suggest four initial topics within this field: benevolent game theory, algorithmic foundations of human rights, mechanistic interpretability of agency representation in neural networks, and reinforcement learning from internal states. While this paper provides valuable insights into the importance of preserving human agency in AI systems, it also acknowledges several limitations such as adding agency preservation does not solve intent-alignment issues entirely; evaluating agency preservation for everyone is computationally challenging; safeguarding human agency may hinder AI progress and may not be realistic given current technological capabilities; AIs are unlikely to modify fundamental biological needs or psychological structures of humans. Overall, this paper emphasizes the need to go beyond alignment to human intent and prioritize the preservation of human agency in AI systems. It calls for further research and exploration in the field of agency foundations to ensure safe and responsible development of AI technologies.