# Concrete Problems in AI Safety

**Authors:** Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané  
**arXiv:** [1606.06565](https://arxiv.org/abs/1606.06565)  
**Date:** 2016-06-21  
**Summary page:** https://www.summarizepaper.com/en/paper/1606.06565v1/

## Key points

- Authors discuss potential impacts of rapid progress in machine learning and AI on society
- Focus on the problem of accidents in machine learning systems
- Comprehensive list of five practical research problems related to accident risk
- Wrong objective function: "avoiding side effects" and "avoiding reward hacking"
- Expensive objective function evaluation: "scalable supervision"
- Undesirable behavior during learning process: "safe exploration" and "distributional shift"
- Review previous work and propose research directions to address these challenges
- Emphasize importance of considering safety when developing AI applications
- Aim to enhance safety and reliability of AI systems as they advance

## Summary

In their paper titled "Concrete Problems in AI Safety," authors Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané discuss the potential impacts of rapid progress in machine learning and artificial intelligence (AI) on society. Specifically, they focus on the problem of accidents in machine learning systems. The authors present a comprehensive list of five practical research problems related to accident risk. These problems are categorized based on their origins: having the wrong objective function ("avoiding side effects" and "avoiding reward hacking"), an objective function that is too expensive to evaluate frequently ("scalable supervision"), or undesirable behavior during the learning process ("safe exploration" and "distributional shift"). To address these challenges, the authors review previous work in these areas and propose research directions that are relevant to cutting-edge AI systems. They emphasize the importance of considering safety when developing forward-looking applications of AI. Overall, this paper provides valuable insights into the potential risks associated with AI technologies and offers practical solutions for mitigating accidents in machine learning systems. The authors' research directions aim to enhance the safety and reliability of AI systems as they continue to advance.