Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?
Authors: Junhua Tang, Zhengsong Zhang. Haipeng Cai
Abstract
Static application security testing (SAST) tools are often deployed on the assumption that detectors configured today will remain effective on future code, including previously unknown vulnerabilities. We test this assumption using 20,000 vulnerable and fixed C/C++ cases from 1,116 open-source projects (2016-2025), evaluated with three CodeQL releases, three Infer releases, and one Semgrep configuration. To distinguish temporal effects from shifts in code composition, we compare yearly populations, project- and label-matched year pairs, and within-project trajectories. We investigate the largest within-project declines through repair diffs, detection predicates, alert locations, and execution logs, while accounting for failed, degraded, and costly analyses. Under changed-file case-label scoring, overall F1 ranges from 0.066 to 0.478, but count-based AUROC stays near chance (0.495-0.526). Newer CodeQL releases attain higher F1 alongside nearly parallel rises in recall and false-positive rate, not stronger separation of vulnerable from fixed cases. In project-matched year pairs, CodeQL R2024 has negative mean later-minus-earlier F1 at every source-year gap, whereas Infer v1.1 and v1.3 have positive means throughout. Among selected drops, only 6 of 39 CodeQL and 2 of 620 Infer findings overlap a changed repair line. For Semgrep, 78.96% of cases show parser or timeout diagnostics even though every run reports success. These results suggest that apparent changes in temporal SAST effectiveness can arise from broader alerting, changing analysis coverage, and incidental findings, not solely from losses in vulnerability detection. Evaluations of temporal sustainability should fix detector conditions, control for projects, inspect repair-level evidence, and report failed or degraded analyses.
The papers around it, by date and closeness
Higher is closer to this paper; left is older. Lines join papers close to each other: groups of lines are sub-topics. Select a point to see why it is there. Back to the summary of this paper
Papers around this one
Ranked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
Longitudinal Analyses of SAST Tools: A CodeQL Case StudyJean-Charles Noirot Ferrand, Kyle Domico, Yohan Beugin et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet
Longitudinal Analyses of SAST Tools measures how CodeQL performance changes across versions on vulnerabilities from open-source repositories.
Abstract
Open-source software (OSS) pipelines rely on automated static analysis tools to prevent the introduction of vulnerabilities in code. However, there is limited understanding of the efficacy of these tools across the OSS ecosystem over time. In this paper, we introduce a novel method to evaluate static application security testing (SAST) tools through longitudinal measurements and perform the largest academic study of CodeQL -- the most prevalent static analysis tool from GitHub -- on OSS codebases. We apply our apparatus on 114 versions of CodeQL over time on 3993 CVEs from 1622 repositories to measure key properties of the tool, culminating in more than 20 billion lines of code analyzed. Fir…
-
Evaluating C/C++ Vulnerability Detectability of Query-Based Static Application Security Testing ToolsZong-Jie Li, Zhibo Liu, Wai-Kin Wong et al. · 2024 · IEEE Transactions on Dependable and Secure Computing · cited 41 times · Not summarized yet
Evaluating C/C++ Vulnerability Detectability compares query-based and conventional SAST tools on C/C++ vulnerability detection.
Abstract
In recent years, query-based static application security testing(Q-SAST) tools such as CodeQL have gained popularity due to their ability to codify vulnerability knowledge into SQL-like queries and search for vulnerabilities in the database derived from the software. The industry has made considerable progress in building Q-SAST tools, facilitating their integration into the continuous integration (CI) pipeline, and sustaining an active community. However, we do not have a systematic understanding of their vulnerability detection capability in comparison to conventional SAST tools. We conduct the first in-depth study of Q-SAST to demystify their C/C++ vulnerability detectability. Our study i…
-
An empirical study on the effectiveness of static C code analyzers for vulnerability detectionStephan Lipp, Sebastian Banescu, A. Pretschner · 2022 · International Symposium on Software Testing and Analysis · cited 180 times · Not summarized yet
An empirical study on the effectiveness of static C code analyzers evaluates static analyzers' ability to detect real-world vulnerabilities in C code.
Abstract
Static code analysis is often used to scan source code for security vulnerabilities. Given the wide range of existing solutions implementing different analysis techniques, it is very challenging to perform an objective comparison between static analysis tools to determine which ones are most effective at detecting vulnerabilities. Existing studies are thereby limited in that (1) they use synthetic datasets, whose vulnerabilities do not reflect the complexity of security bugs that can be found in practice and/or (2) they do not provide differentiated analyses w.r.t. the types of vulnerabilities output by the static analyzers. Hence, their conclusions about an analyzer's capability to detect v…
-
An Extensive Comparison of Static Application Security Testing ToolsMatteo Esposito, Valentina Falaschi, Davide Falessi · 2024 · International Conference on Evaluation & Assessment in Software Engineering · cited 36 times · Not summarized yet
An Extensive Comparison benchmarks SAST tools on a synthetic Java codebase, rather than tracking analyzers across evolving C/C++ projects.
Abstract
Context: Static Application Security Testing Tools (SASTTs) identify software vulnerabilities to support the security and reliability of software applications. Interestingly, several studies have suggested that alternative solutions may be more effective than SASTTs due to their tendency to generate false alarms, commonly referred to as low Precision. Aim: We aim to comprehensively evaluate SASTTs, setting a reliable benchmark for assessing and finding gaps in vulnerability identification mechanisms based on SASTTs or alternatives. Method: Our SASTTs evaluation is based on a controlled, though synthetic, Java codebase. It involves an assessment of 1.5 million test executions, and it features…
-
Large Language Models Versus Static Code Analysis Tools: A Systematic Benchmark for Vulnerability DetectionDamian Gnieciak, Tomasz Szandała · 2025 · IEEE Access · cited 21 times · Not summarized yet
Large Language Models Versus Static Code Analysis Tools compares rule-based analyzers with language models on vulnerabilities in C# projects.
Abstract
Modern software relies on a multitude of automated testing and quality assurance tools to prevent errors, bugs and potential vulnerabilities. This study sets out to provide a head-to-head, quantitative and qualitative evaluation of six automated approaches: three industry-standard rule-based static code-analysis tools (SonarQube, CodeQL and SnykCode) and three state-of-the-art large language models hosted on the GitHub Models platform (GPT-4.1, Mistral Large and DeepSeek V3). Using a curated suite of ten real-world C# projects that embed 63 vulnerabilities across common categories such as SQL injection, hard-coded secrets and outdated dependencies, we measure classical detection accuracy (pr…
-
Finding Fixed Vulnerabilities with Off-the-Shelf Static AnalysisT. Dunlap, Seaver Thorn, W. Enck et al. · 2023 · European Symposium on Security and Privacy · cited 17 times · Not summarized yet
Finding Fixed Vulnerabilities applies static security analysis to identify vulnerability fixes by comparing alerts across project versions.
Abstract
Software depends on upstream projects that regularly fix vulnerabilities, but the documentation of those vulnerabilities is often unreliable or unavailable. Automating the collection of existing vulnerability fixes is essential for downstream projects to reliably update their dependencies due to the sheer number of dependencies in modern software. Prior efforts rely solely on incomplete databases or imprecise or inaccurate statistical analysis of upstream repositories. In this paper, we introduce Differential Alert Analysis (DAA) to discover vulnerability fixes in software projects. In contrast to statistical analysis, DAA leverages static analysis security testing (SAST) tools, which reason…
-
Persistent Human Feedback, LLMs, and Static Analyzers for Secure Code Generation and Vulnerability DetectionEhsan Firouzi, Mohammad Ghafari · 2026 · IEEE International Conference on Software Analysis, Evolution, and Reengineering · cited 8 times · Not summarized yet
Persistent Human Feedback evaluates CodeQL and Semgrep's vulnerability detection against human-validated LLM-generated code.
Abstract
Existing literature heavily relies on static analysis tools to evaluate LLMs for secure code generation and vulnerability detection. We reviewed 1,080 LLM-generated code samples, built a human-validated ground-truth, and compared the outputs of two widely used static security tools, CodeQL and Semgrep, against this corpus. While 61% of the samples were genuinely secure, Semgrep and CodeQL classified 60% and 80% as secure, respectively. Despite the apparent agreement in aggregate statistics, per-sample analysis reveals substantial discrepancies: only 65% of Semgrep's and 61% of CodeQL's reports correctly matched the ground truth. These results question the reliability of static analysis tools…
-
SmartCode AI: A Hybrid Static Analysis and Ensemble Transformer Framework for Secure Code Review and Automated RemediationN. Maheswari, A. K. Prabhakaran, R. Vignesh et al. · 2026 · International Congress on Information and Communication Technology · Not summarized yet
SmartCode AI combines static analysis with transformer models for secure code review and automated remediation.
Abstract
Application security vulnerabilities continue to dominate the threat landscape, contributing to nearly seventy percent of reported security breaches. The growing volume of software applications, combined with increasing architectural complexity, has significantly expanded the attack surface. Traditional Static Application Security Testing (SAST) tools remain widely adopted; however, existing approaches suffer from critical limitations. High false positive rates, often ranging between sixty and eighty percent, create alert fatigue and reduce developer trust. Pattern-based detection mechanisms lack contextual understanding of business logic, leading to misclassification of safe code as vulnera…
Prior work it builds on
-
Comparison and Evaluation on Static Application Security Testing (SAST) Tools for JavaKai-Xuan Li, Sen Chen, Ling-Ling Fan et al. · 2023 · ESEC/SIGSOFT FSE · cited 86 times · Not summarized yet
Comparison and Evaluation compares Java SAST tools on effectiveness, consistency, and performance using synthetic and real-world benchmarks.
Abstract
Static application security testing (SAST) takes a significant role in the software development life cycle (SDLC). However, it is challenging to comprehensively evaluate the effectiveness of SAST tools to determine which is the better one for detecting vulnerabilities. In this paper, based on well-defined criteria, we first selected seven free or open-source SAST tools from 161 existing tools for further evaluation. Owing to the synthetic and newly-constructed real-world benchmarks, we evaluated and compared these SAST tools from different and comprehensive perspectives such as effectiveness, consistency, and performance. While SAST tools perform well on synthetic benchmarks, our results ind…
-
How Many of All Bugs Do We Find? A Study of Static Bug DetectorsA. Habib, Michael Pradel · 2018 · International Conference on Automated Software Engineering · cited 130 times · Not summarized yet
How Many of All Bugs Do We Find? measures static detectors' coverage of known Java bugs, rather than security vulnerabilities over time.
Abstract
Static bug detectors are becoming increasingly popular and are widely used by professional software developers. While most work on bug detectors focuses on whether they find bugs at all, and on how many false positives they report in addition to legitimate warnings, the inverse question is often neglected: How many of all real-world bugs do static bug detectors find? This paper addresses this question by studying the results of applying three widely used static bug detectors to an extended version of the Defects4J dataset that consists of 15 Java projects with 594 known bugs. To decide which of these bugs the tools detect, we use a novel methodology that combines an automatic analysis of war…
-
An empirical study of security warnings from static application security testing toolsBushra Aloraini, M. Nagappan, D. Germán et al. · 2019 · Journal of Systems and Software · cited 66 times · Not summarized yet
An empirical study of security warnings investigates false positives across SAST warning types and their evolution over time.
Abstract
Abstract The Open Web Application Security Project (OWASP) defines Static Application Security Testing (SAST) tools as those that can help find security vulnerabilities in the source code or compiled code of software. Such tools detect and classify the vulnerability warnings into one of many types (e.g., input validation and representation). It is well known that these tools produce high numbers of false positive warnings. However, what is not known is if specific types of warnings have a higher predisposition to be false positives or not. Therefore, our goal is to investigate the different types of SAST-produced warnings and their evolution over time to determine if one type of warning is m…
-
Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source SoftwareM. Beller, R. Bholanath, Shane McIntosh et al. · 2016 · IEEE International Conference on Software Analysis, Evolution, and Reengineering · cited 234 times · Not summarized yet
Analyzing the State of Static Analysis evaluates static analysis at scale in open-source software.
-
Do Developers Use Static Application Security Testing (SAST) Tools Straight Out of the Box? A large-scale Empirical StudyGareth Bennett, Tracy Hall, Steve Counsell et al. · 2024 · International Symposium on Empirical Software Engineering and Measurement · cited 13 times · Not summarized yet
Do Developers Use Static Application Security Testing Tools Straight Out of the Box? studies how developers configure and combine SAST tools.
Abstract
Static application Security Testing (SAST) tools are an established means of detecting vulnerabilities early in development. Previous studies have reported low detection rates from SAST tools and recommend either combining SAST tools or configuring rule sets to detect more vulnerabilities. However, while previous work suggests that developers rarely combine or configure any of the Automatic Static Analysis Tools (ASATs) they use, it is currently unclear whether SAST tools are used directly “out of the box”. To understand how developers use SAST tools, we performed a large-scale survey involving 1,263 developers. We pre-screened developers to establish their SAST use and found that only 20% (…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 9 November 2026.