Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

Junhua Tang; Zhengsong Zhang. Haipeng Cai

AI summary · arXiv: 2611.03742v1 · cs.CR

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

Junhua Tang, Zhengsong Zhang. Haipeng Cai

Original paper Submitted to arXiv on 02 Nov. 2026 License: NONEXCLUSIVE-DISTRIB 1.0 Also in: fr

How to read this summary? Illustrated guide

The essentials

Temporal changes in measured SAST performance reflect broader alerting and changing analysis coverage as well as vulnerability detection, underscoring the need for controlled evaluations.

Based on the abstract only: check the details in the original paper.

Question

Do static security analyzers remain effective as C/C++ code evolves over time, including for vulnerabilities not previously seen? The study also asks whether apparent temporal changes reflect detector performance or shifts in projects, code composition, and analysis coverage.

Method

The authors evaluate 20,000 vulnerable and fixed cases from 1,116 open-source projects spanning 2016–2025, using three CodeQL releases, three Infer releases, and one Semgrep configuration. They compare yearly populations, project- and label-matched year pairs, and within-project trajectories; for selected declines, they inspect repair diffs, detection predicates, alert locations, and execution logs, while accounting for failed, degraded, and costly analyses.

Findings

Result Under changed-file case-label scoring, F1 ranges from 0.066 to 0.478, while count-based AUROC remains near chance at 0.495–0.526.
See the passage in the paper
Under changed-file case-label scoring, overall F1 ranges from 0.066 to 0.478, but count-based AUROC stays near chance (0.495-0.526).
Authors’ interpretation Newer CodeQL releases have higher F1, accompanied by nearly parallel increases in recall and false-positive rate rather than stronger separation of vulnerable and fixed cases.
See the passage in the paper
Newer CodeQL releases attain higher F1 alongside nearly parallel rises in recall and false-positive rate, not stronger separation of vulnerable from fixed cases.
Result In project-matched year pairs, CodeQL R2024 has negative mean later-minus-earlier F1 at every source-year gap, whereas Infer v1.1 and v1.3 have positive means throughout.
See the passage in the paper
In project-matched year pairs, CodeQL R2024 has negative mean later-minus-earlier F1 at every source-year gap, whereas Infer v1.1 and v1.3 have positive means throughout.
Result Few selected analyzer findings overlap changed repair lines, and Semgrep often reports parser or timeout diagnostics despite successful run status.
See the passage in the paper
Among selected drops, only 6 of 39 CodeQL and 2 of 620 Infer findings overlap a changed repair line. For Semgrep, 78.96% of cases show parser or timeout diagnostics even though every run reports success.
Authors’ interpretation The authors argue that observed temporal changes can reflect broader alerting, changing analysis coverage, and incidental findings, not just declining vulnerability detection.
See the passage in the paper
These results suggest that apparent changes in temporal SAST effectiveness can arise from broader alerting, changing analysis coverage, and incidental findings, not solely from losses in vulnerability detection.

Limitations

The evaluation is scoped to the studied C/C++ cases, open-source projects, analyzer releases, and Semgrep configuration, so its results do not establish performance beyond these conditions.

Why it matters

The authors argue that evaluations of temporal SAST sustainability should hold detector conditions fixed, control for project differences, inspect repair-level evidence, and report failed or degraded analyses. This reframes apparent changes in effectiveness as potentially involving alert breadth and analysis coverage as well as detection.

Written by AI from the paper. Passages are checked word for word against the text we analysed; the rest may contain errors.

AI-generated keywords: static application security testing temporal evaluation C and C++ vulnerability detection alert coverage

Read further

Comprehensive Summary
Layman's Summary
Blog article
Original abstract

The paper asks whether static application security testing (SAST) tools configured at a given point in time remain effective as C and C++ code evolves, including when they encounter vulnerabilities they may not have seen before. The authors study 20,000 vulnerable and fixed cases from 1,116 open-source projects spanning 2016–2025. This summary is based only on the paper’s title and abstract; the full paper was unavailable, so details beyond those sources cannot be verified. The evaluation covers three releases of CodeQL, three releases of Infer, and one Semgrep configuration. To separate changes over time from changes in the kinds of code being tested, the authors compare yearly case populations, match cases by project and label across years, and track trajectories within projects. They also investigate selected large declines by examining repair diffs, detector predicates, alert locations, and execution logs. The abstract states that the study accounts for analyses that fail, degrade, or incur high costs. Its scoring approach labels cases using changed files and measures both F1 and count-based AUROC. Across the evaluated tools and settings, F1 ranges from 0.066 to 0.478, while count-based AUROC remains close to chance, between 0.495 and 0.526. This contrast is important: F1 can vary with how readily a tool reports alerts, while an AUROC near 0.5 indicates little separation between vulnerable and fixed cases under the count-based measure. Newer CodeQL releases achieve higher F1 scores, but the authors attribute this pattern to recall and false-positive rates increasing in nearly parallel, rather than to improved discrimination between vulnerable and fixed cases. Results also vary by tool and comparison method. In project-matched comparisons across years, CodeQL R2024 has a negative mean change in F1 for every source-year gap examined. Infer v1.1 and v1.3, by contrast, show positive mean changes in all those comparisons. These findings suggest that no single overall temporal trend describes all tools; results depend on both the metric and how cases are matched. The abstract does not report the sizes of these year-pair effects or provide enough detail to assess their statistical uncertainty. Inspection of selected declines raises questions about whether alerts consistently identify the code changes that repair vulnerabilities. Among the examined cases, only 6 of 39 CodeQL findings and 2 of 620 Infer findings overlap with a changed repair line. For Semgrep, 78.96% of cases have parser or timeout diagnostics, even though every run is reported as successful. A nominally successful execution therefore may not mean that analysis proceeded cleanly or adequately covered the code. The authors conclude that apparent temporal changes in SAST effectiveness may reflect broader alerting, shifting analysis coverage, or incidental findings—not simply gains or losses in vulnerability-detection ability. They recommend keeping detector conditions fixed in temporal evaluations, controlling for project differences, checking alerts against repair-level evidence, and explicitly reporting failed or degraded analyses. The available account has important limitations because it includes only the abstract. It does not provide the detailed dataset construction, tool configurations, full metric definitions, uncertainty estimates, or broader caveats needed to assess the results independently. The analysis also concerns a specific collection of open-source C and C++ cases and tools, so the abstract alone cannot establish how well the conclusions generalize beyond them.

Do Static Security Scanners Keep Up as C and C++ Code Changes?

A security scanner that works well today may not work as well on tomorrow’s code. That is the central question of “Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?” by Junhua Tang, Zhengsong Zhang, and Haipeng Cai. The study examines whether static application security testing (SAST) tools, configured at a particular point in time, continue to distinguish vulnerable code from repaired code as software evolves.

Based on the paper’s title and abstract, the researchers evaluated 20,000 vulnerable and fixed cases drawn from 1,116 open-source C and C++ projects spanning 2016 to 2025. They tested three releases of CodeQL, three releases of Infer, and one Semgrep configuration. Because the full paper was unavailable, the details below are limited to what the abstract reports; aspects such as dataset construction and exact tool settings cannot be independently verified.

How the Researchers Compared Tools Over Time

A simple comparison between old and new code can be misleading. If the cases from one year involve different projects or kinds of vulnerabilities than those from another, a change in scores might reflect the sample rather than a change in the tools. The study therefore used several comparison approaches: examining yearly case populations, matching cases across years by project and label, and tracking results within projects over time.

The authors also inspected selected cases where performance declined. They looked at repair diffs, detector conditions, alert locations, and execution logs—evidence that can help distinguish a meaningful detection from an alert that merely happens to appear near a vulnerable case. The abstract says their evaluation also accounts for analyses that fail, degrade, or become expensive.

For scoring, cases were labeled using changed files. The researchers report both F1 and count-based area under the receiver operating characteristic curve (AUROC). In broad terms, F1 summarizes a tool’s precision and recall at a chosen reporting threshold, while AUROC measures how well a score separates two classes across thresholds. The abstract does not provide the full metric definitions.

Higher F1 Did Not Mean Better Separation

Across the tools and settings, F1 scores ranged from 0.066 to 0.478. Count-based AUROC, however, stayed close to chance, between 0.495 and 0.526. That contrast is the study’s most important caution: a tool can report more alerts and receive a different F1 score without becoming better at distinguishing vulnerable cases from fixed ones.

Newer CodeQL releases achieved higher F1 scores. But the authors attribute this pattern to recall and false-positive rates rising in nearly parallel, rather than to improved discrimination. In other words, CodeQL may have caught more vulnerable cases while also producing more alerts on fixed cases. A higher F1 should therefore not be read on its own as evidence that the detector has become more reliable.

The time trends also differed by tool and comparison method. In project-matched comparisons, CodeQL R2024 had a negative mean change in F1 for every source-year gap examined. Infer v1.1 and v1.3, by contrast, had positive mean changes in all those comparisons. The abstract does not report the sizes of these effects or enough information to judge their statistical uncertainty.

Alerts Need to Be Checked Against Repairs

One way to test whether an alert is connected to a vulnerability is to see whether it overlaps with the code changed to repair that vulnerability. In selected cases examined by the authors, only 6 of 39 CodeQL findings and 2 of 620 Infer findings overlapped with a changed repair line. These figures raise questions about how consistently alerts point to the code directly involved in a fix. They should be understood as results from the examined cases, not necessarily as rates across the entire dataset.

The Semgrep results highlight a separate challenge: analysis quality. The abstract reports parser or timeout diagnostics in 78.96% of cases, even though every run was labeled successful. That gap shows why a successful process exit may not be enough to establish that a tool analyzed the intended code cleanly or completely.

What the Findings Mean for Security Evaluations

The study argues that apparent changes in SAST performance over time can have several explanations. Tools may issue alerts more broadly, analysis coverage may shift, or findings may be incidental to the actual repair. A simple score comparison cannot always tell these possibilities apart.

The authors recommend several practices for researchers evaluating security tools over time:

Keep detector conditions fixed when comparing results across periods.
Control for differences between projects and case populations.
Check alerts against repair-level evidence where possible.
Report failed or degraded analyses, not only runs marked successful.

The findings should also be read within their limits. The study concerns a specific collection of open-source C and C++ cases and selected tools. The abstract alone does not provide the detailed configurations, uncertainty estimates, or broader caveats needed to assess the results independently, nor can it establish whether the conclusions generalize to other languages or settings.

Even with those limits, the paper offers a useful reminder: a newer scanner or a higher headline score does not automatically mean better vulnerability detection. To understand whether a SAST tool has kept pace with evolving code, evaluators need to examine not just how many alerts it produces, but whether those alerts separate vulnerable from fixed code, point to meaningful repairs, and come from analyses that actually ran as intended.

Created on 09 Nov. 2026

Was this synthesis useful?

Score: 0

Ask questions about this paper to our AI assistant

How to chat with this paper? Illustrated guide

Compare with the PDF

You can also chat with multiple papers at once here.

AI assistant instructions?

Tools for this paper

Explain it differently 1 free rewrite left today

Save to a collection Group papers in collections to compare them, chat with them and export them. Continue with Google Create a free account Log in

Export BibTeX · RIS (Zotero, Mendeley, EndNote) · Obsidian (Markdown) · Notion (Markdown)

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero

Citation alerts Citation alerts guide

Who works on it? The people and institutions who publish on its topic

Weekly letter Presented in "Computing beyond AI this week" of 9 November 2026. Computing beyond AI this week: What noise reveals, and what guarantees conceal

Journal club Premium+ Slides to present it Propose for the journal club

Versions What changed between versions

Talk about it in public Premium+ Press kit: release, questions, hype check Press kit guide

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

The essentials

Question

Method

Findings

Limitations

Why it matters

Read further

Definitions

Do Static Security Scanners Keep Up as C and C++ Code Changes?

How the Researchers Compared Tools Over Time

Higher F1 Did Not Mean Better Separation

Alerts Need to Be Checked Against Repairs

What the Findings Mean for Security Evaluations

Ask questions about this paper to our AI assistant

Tools for this paper

Advanced tools Premium

Papers around this one

Closest papers

Prior work it builds on

Go further

Cite this paper (BibTeX)