# Carbon Filter: Real-time Alert Triage Using Large Scale Clustering and Fast Search

**Authors:** Jonathan Oliver, Raghav Batta, Adam Bates, Muhammad Adil Inam, Shelly Mehta, Shugao Xia  
**arXiv:** [2405.04691](https://arxiv.org/abs/2405.04691)  
**Date:** 2024-05-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.04691v1/

## Key points

- "Alert fatigue" is a significant issue in Security Operations Centers (SOCs)
- Endpoint detection products generate a high number of false alerts due to pattern matching against behavioral rules
- Alert triage techniques leveraging data provenance are not practical for real-world production environments
- Carbon Filter is a system developed by a team comprising Jonathan Oliver, Raghav Batta, Adam Bates, Muhammad Adil Inam, Shelly Mehta, and Shugao Xia
- Carbon Filter uses statistical learning principles to efficiently discern false alert triggers from suspicious activities by analyzing the context surrounding process initiation
- Leveraging fast-search algorithms allows Carbon Filter to scale seamlessly to handle millions of alerts daily
- The model can process up to 20 million alerts per hour when queries are batched
- Carbon Filter delivers a six-fold enhancement in Signal-to-Noise ratio without compromising alert triage performance
- This system significantly reduces manual review burden on analysts and ensures prompt identification of genuine security threats among the deluge of alerts in modern SOC environments

## Summary

In today's Security Operations Center (SOC) landscape, the issue of "alert fatigue" looms large. Analysts often spend a significant amount of time sifting through false alerts generated by endpoint detection products. These products use pattern matching against behavioral rules, resulting in a high number of false positives that divert attention from genuine threats. While alert triage techniques leveraging data provenance hold promise, they are not practical for real-world production environments due to their time-consuming nature and the sheer volume faced by EDR customers. To address this pressing issue, a team comprising Jonathan Oliver, Raghav Batta, Adam Bates, Muhammad Adil Inam, Shelly Mehta, and Shugao Xia introduces Carbon Filter—a cutting-edge system grounded in statistical learning principles that revolutionizes the manual review process for analysts. The key insight driving Carbon Filter's efficacy lies in its ability to efficiently discern false alert triggers from suspicious activities by analyzing the context surrounding process initiation (e.g., command lines) responsible for launching said processes. Leveraging fast-search algorithms for both training and inference tasks enables Carbon Filter to seamlessly scale to handle millions of alerts daily. By batching queries to the model, the team observes an impressive theoretical maximum throughput of 20 million alerts per hour. Drawing on extensive analysis of tens of millions of alerts sourced from customer deployments, Carbon Filter delivers a remarkable six-fold enhancement in Signal-to-Noise ratio without compromising alert triage performance. This breakthrough not only significantly reduces manual review burden on analysts but also ensures that genuine security threats are promptly identified and addressed amidst the deluge of alerts inundating modern SOC environments. With its unparalleled scalability and efficiency in separating noise from actionable signals, Carbon Filter stands as a beacon of innovation in the realm of real-time alert triage solutions—a game-changer poised to elevate SOC operations to new heights of effectiveness and responsiveness in combating cyber threats.