# Taming Uncertainty via Automation: Observing, Analyzing, and Optimizing Agentic AI Systems

**Authors:** Dany Moshkovich, Sergey Zeltyn  
**arXiv:** [2507.11277](https://arxiv.org/abs/2507.11277)  
**Date:** 2025-07-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2507.11277v1/

## Key points

- Large Language Models (LLMs) are prevalent in agentic systems
- Systems powered by LLMs carry out complex, adaptive workflows using memory, tools, and dynamic planning
- Challenges include probabilistic reasoning, evolving memory states, and flexible execution paths
- Conventional software observability practices are inadequate for these systems
- AgentOps framework is introduced to observe, analyze, optimize, and automate the operation of agentic AI systems
- Framework caters to developers, testers, SREs, and business users at different stages of the system's lifecycle
- Automation Pipeline consists of six stages aimed at enhancing system performance
- Emphasis on automation managing uncertainty within AI systems for safe and adaptive operation
- Automation tames uncertainty rather than eliminating it entirely for improved performance and reliability

## Summary

In the realm of artificial intelligence, Large Language Models (LLMs) are becoming increasingly prevalent in agentic systems. These systems consist of interacting agents powered by LLMs that carry out complex, adaptive workflows utilizing memory, tools, and dynamic planning. While these systems offer remarkable capabilities, they also introduce a unique set of challenges due to probabilistic reasoning, evolving memory states, and flexible execution paths. Conventional software observability and operational practices are inadequate in addressing these complexities. To tackle these challenges, this paper introduces AgentOps: a comprehensive framework designed to observe, analyze, optimize, and automate the operation of agentic AI systems. The framework recognizes the distinct needs of four key roles within the system's lifecycle: developers, testers, site reliability engineers (SREs), and business users. Each role interacts with the system at different stages and contributes to its overall functionality. Central to the AgentOps framework is the Automation Pipeline, which consists of six stages aimed at enhancing system performance. These stages include behavior observation, metric collection, issue detection, root cause analysis, optimized recommendations,and runtime automation. Throughout this process,

emphasis is placed on the pivotal role of automation in managing uncertainty within AI systems. Rather than eliminating uncertainty entirely,

automation serves to tame it effectively to ensure safe and adaptive operation. Authored by Dany Moshkovich and Sergey Zeltyn,this paper sheds light on how observing uncertainty through automation can lead to improved performance and reliability in agentic AI systems.By implementing the principles outlined in AgentOps,

organizations can navigate the intricate landscape of AI operations with greater confidence and efficiency.