# BotzoneBench: Scalable LLM Evaluation via Graded AI Anchors

**Authors:** Lingfeng Li, Yunlong Lu, Yuefei Zhang, Jingyu Yao, Yixin Zhu, KeYuan Cheng, Yongyi Wang, Qirui Zheng, Xionghui Yang, Wenxin Li  
**arXiv:** [2602.13214](https://arxiv.org/abs/2602.13214)  
**Date:** 2026-01-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2602.13214v1/

## Key points

- Large Language Models (LLMs) are increasingly used in interactive environments for strategic decision-making in artificial intelligence (AI).
- Traditional benchmarks for LLMs focus on static reasoning abilities and do not capture the dynamic nature of strategic decision-making.
- LLM-vs-LLM tournaments in game-based settings have been introduced to evaluate strategic capabilities, but they come with high computational costs and lack stable performance benchmarks.
- BotzoneBench is a new evaluation method that anchors LLM assessment to fixed hierarchies of skill-calibrated game AI, enabling linear-time absolute skill measurement with stable cross-temporal interpretability.
- BotzoneBench evaluates LLMs across eight diverse games and has uncovered significant performance variations among different models, demonstrating proficiency levels comparable to specialized game AI.
- This evaluation paradigm can be applied beyond gaming applications to any domain with well-defined skill hierarchies, representing a significant advancement in AI evaluation methodology.

## Summary

In the realm of artificial intelligence (AI), Large Language Models (LLMs) are becoming increasingly prevalent in interactive environments that demand strategic decision-making. However, evaluating the strategic capabilities of these models in a systematic manner poses a significant challenge. Traditional benchmarks for LLMs typically focus on assessing static reasoning abilities through isolated tasks, failing to capture the dynamic nature of strategic decision-making. Recent evaluations have attempted to address this by organizing LLM-vs-LLM tournaments in game-based settings. While these tournaments provide relative rankings based on transient model pools, they come with high computational costs and lack stable performance benchmarks for long-term tracking. To overcome these limitations, a scalable evaluation framework is needed that can measure LLM strategic reasoning against consistent and interpretable standards rather than relying on fluctuating peer models. In response to this challenge, a new approach called BotzoneBench has been introduced. This evaluation method anchors LLM assessment to fixed hierarchies of skill-calibrated game Artificial Intelligence (AI), enabling linear-time absolute skill measurement with stable cross-temporal interpretability. BotzoneBench operates on the established competitive infrastructure of the Botzone platform and evaluates LLMs across eight diverse games ranging from deterministic perfect-information board games to stochastic imperfect-information card games. Through a thorough analysis of 177,047 state-action pairs from five leading models, significant performance variations were uncovered, highlighting distinct strategic behaviors exhibited by different models. The top-performing LLMs demonstrated proficiency levels comparable to mid-to-high-tier specialized game AI across multiple domains. This anchored evaluation paradigm extends beyond gaming applications and can be applied to any domain with well-defined skill hierarchies. By establishing a scalable and reusable framework for assessing interactive AI capabilities, BotzoneBench represents a significant advancement in the field of AI evaluation methodology.