# Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps

**Authors:** Alankrit Chona, Igor Kozlov, Ambuj Kumar  
**arXiv:** [2604.19533](https://arxiv.org/abs/2604.19533)  
**Date:** 2026-04-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2604.19533v1/

## Key points

- The Cyber Defense Benchmark introduces a new method for evaluating the performance of large language model (LLM) agents in cybersecurity operations (SecOps).
- The benchmark focuses on assessing how well LLM agents can handle threat hunting within a security operations center (SOC) environment.
- It challenges LLM agents with raw Windows event logs and incorporates 106 real attack procedures from the OTRF Security-Datasets corpus, covering 86 MITRE ATT&CK sub-techniques across 12 tactics.
- Evaluation of five frontier models reveals significant shortcomings in their performance, with even the best-performing model only correctly flagging approximately 3.8% of malicious events on average.
- A passing score for unsupervised SOC deployment requires a minimum recall rate of >=50% on every ATT&CK tactic, but none of the models meet this criterion.
- Current LLMs are not equipped for open-ended threat hunting tasks based on evidence-driven approaches despite strong performance in curated Q&A security benchmarks.
- The Cyber Defense Benchmark highlights limitations and challenges faced by LLM agents in real-world cybersecurity scenarios, emphasizing the need for further advancements to improve their performance.

## Summary

The Cyber Defense Benchmark introduces a new method for evaluating the performance of large language model (LLM) agents in the field of cybersecurity operations (SecOps). This benchmark focuses on assessing how well LLM agents can handle the core task of threat hunting within a security operations center (SOC) environment. The benchmark challenges LLM agents by presenting them with a database of raw Windows event logs without any guided questions or hints. It incorporates 106 real attack procedures from the OTRF Security-Datasets corpus, covering 86 MITRE ATT&CK sub-techniques across 12 tactics. Each episode presents the agent with an in-memory SQLite database containing obfuscated and time-shifted log records from a deterministic campaign simulator. The agent must use SQL queries to uncover malicious event timestamps and flag them according to Sigma-rule-derived ground truth. Evaluation of five frontier models - Claude Opus 4.6, GPT-5, Gemini 3.1 Pro, Kimi K2.5, and Gemini 3 Flash - reveals significant shortcomings in their performance on 26 campaigns covering 105 out of 106 procedures. Even the best-performing model (Claude Opus 4.6) only correctly flags approximately 3.8% of malicious events on average, with no model successfully identifying all flags across any run. To establish a passing score for unsupervised SOC deployment, a minimum recall rate of >=50% on every ATT&CK tactic is set as the bar; however, none of the models meet this criterion. While the leader surpasses this threshold for five out of thirteen tactics, it fails to do so for the remaining eight tactics. These findings highlight that current LLMs are not equipped for open-ended threat hunting tasks based on evidence-driven approaches despite their strong performance in curated Q&A security benchmarks. The Cyber Defense Benchmark provides valuable insights into the limitations and challenges faced by LLM agents in real-world cybersecurity scenarios. It underscores the need for further advancements in this area to improve their performance.