Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?
Auteurs : Junhua Tang, Zhengsong Zhang. Haipeng Cai
Résumé scientifique
Static application security testing (SAST) tools are often deployed on the assumption that detectors configured today will remain effective on future code, including previously unknown vulnerabilities. We test this assumption using 20,000 vulnerable and fixed C/C++ cases from 1,116 open-source projects (2016-2025), evaluated with three CodeQL releases, three Infer releases, and one Semgrep configuration. To distinguish temporal effects from shifts in code composition, we compare yearly populations, project- and label-matched year pairs, and within-project trajectories. We investigate the largest within-project declines through repair diffs, detection predicates, alert locations, and execution logs, while accounting for failed, degraded, and costly analyses. Under changed-file case-label scoring, overall F1 ranges from 0.066 to 0.478, but count-based AUROC stays near chance (0.495-0.526). Newer CodeQL releases attain higher F1 alongside nearly parallel rises in recall and false-positive rate, not stronger separation of vulnerable from fixed cases. In project-matched year pairs, CodeQL R2024 has negative mean later-minus-earlier F1 at every source-year gap, whereas Infer v1.1 and v1.3 have positive means throughout. Among selected drops, only 6 of 39 CodeQL and 2 of 620 Infer findings overlap a changed repair line. For Semgrep, 78.96% of cases show parser or timeout diagnostics even though every run reports success. These results suggest that apparent changes in temporal SAST effectiveness can arise from broader alerting, changing analysis coverage, and incidental findings, not solely from losses in vulnerability detection. Evaluations of temporal sustainability should fix detector conditions, control for projects, inspect repair-level evidence, and report failed or degraded analyses.
Les articles autour de lui, par date et par proximité
Plus haut, c’est plus proche de cet article ; à gauche, c’est plus ancien. Les traits relient les articles proches entre eux : les groupes de traits sont des sous-thèmes. Sélectionnez un point pour voir pourquoi il est là. Retour au résumé de cet article
Les articles autour de celui-ci
Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.
Les articles les plus proches
-
Longitudinal Analyses of SAST Tools: A CodeQL Case StudyJean-Charles Noirot Ferrand, Kyle Domico, Yohan Beugin et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 1 fois · Pas encore résumé
Longitudinal Analyses of SAST Tools étudie l’efficacité de CodeQL sur des dépôts open source au fil du temps, comme cette étude sur la stabilité temporelle des analyseurs SAST.
Résumé scientifique
Open-source software (OSS) pipelines rely on automated static analysis tools to prevent the introduction of vulnerabilities in code. However, there is limited understanding of the efficacy of these tools across the OSS ecosystem over time. In this paper, we introduce a novel method to evaluate static application security testing (SAST) tools through longitudinal measurements and perform the largest academic study of CodeQL -- the most prevalent static analysis tool from GitHub -- on OSS codebases. We apply our apparatus on 114 versions of CodeQL over time on 3993 CVEs from 1622 repositories to measure key properties of the tool, culminating in more than 20 billion lines of code analyzed. Fir…
-
Evaluating C/C++ Vulnerability Detectability of Query-Based Static Application Security Testing ToolsZong-Jie Li, Zhibo Liu, Wai-Kin Wong et al. · 2024 · IEEE Transactions on Dependable and Secure Computing · cité 41 fois · Pas encore résumé
Evaluating C/C++ Vulnerability Detectability compare la détection de vulnérabilités C/C++ par des outils Q-SAST et des analyseurs conventionnels, plutôt que leur évolution temporelle.
Résumé scientifique
In recent years, query-based static application security testing(Q-SAST) tools such as CodeQL have gained popularity due to their ability to codify vulnerability knowledge into SQL-like queries and search for vulnerabilities in the database derived from the software. The industry has made considerable progress in building Q-SAST tools, facilitating their integration into the continuous integration (CI) pipeline, and sustaining an active community. However, we do not have a systematic understanding of their vulnerability detection capability in comparison to conventional SAST tools. We conduct the first in-depth study of Q-SAST to demystify their C/C++ vulnerability detectability. Our study i…
-
An empirical study on the effectiveness of static C code analyzers for vulnerability detectionStephan Lipp, Sebastian Banescu, A. Pretschner · 2022 · International Symposium on Software Testing and Analysis · cité 180 fois · Pas encore résumé
An empirical study on the effectiveness of static C code analyzers compare des analyseurs statiques pour détecter des vulnérabilités en C et est cité par cette étude.
Résumé scientifique
Static code analysis is often used to scan source code for security vulnerabilities. Given the wide range of existing solutions implementing different analysis techniques, it is very challenging to perform an objective comparison between static analysis tools to determine which ones are most effective at detecting vulnerabilities. Existing studies are thereby limited in that (1) they use synthetic datasets, whose vulnerabilities do not reflect the complexity of security bugs that can be found in practice and/or (2) they do not provide differentiated analyses w.r.t. the types of vulnerabilities output by the static analyzers. Hence, their conclusions about an analyzer's capability to detect v…
-
An Extensive Comparison of Static Application Security Testing ToolsMatteo Esposito, Valentina Falaschi, Davide Falessi · 2024 · International Conference on Evaluation & Assessment in Software Engineering · cité 36 fois · Pas encore résumé
An Extensive Comparison of Static Application Security Testing Tools évalue des outils SAST sur un corpus synthétique Java, apportant un contexte comparatif hors C/C++.
Résumé scientifique
Context: Static Application Security Testing Tools (SASTTs) identify software vulnerabilities to support the security and reliability of software applications. Interestingly, several studies have suggested that alternative solutions may be more effective than SASTTs due to their tendency to generate false alarms, commonly referred to as low Precision. Aim: We aim to comprehensively evaluate SASTTs, setting a reliable benchmark for assessing and finding gaps in vulnerability identification mechanisms based on SASTTs or alternatives. Method: Our SASTTs evaluation is based on a controlled, though synthetic, Java codebase. It involves an assessment of 1.5 million test executions, and it features…
-
Large Language Models Versus Static Code Analysis Tools: A Systematic Benchmark for Vulnerability DetectionDamian Gnieciak, Tomasz Szandała · 2025 · IEEE Access · cité 21 fois · Pas encore résumé
Large Language Models Versus Static Code Analysis Tools compare des analyseurs statiques et des modèles de langage sur des vulnérabilités C#, un contexte différent de l’évolution C/C++.
Résumé scientifique
Modern software relies on a multitude of automated testing and quality assurance tools to prevent errors, bugs and potential vulnerabilities. This study sets out to provide a head-to-head, quantitative and qualitative evaluation of six automated approaches: three industry-standard rule-based static code-analysis tools (SonarQube, CodeQL and SnykCode) and three state-of-the-art large language models hosted on the GitHub Models platform (GPT-4.1, Mistral Large and DeepSeek V3). Using a curated suite of ten real-world C# projects that embed 63 vulnerabilities across common categories such as SQL injection, hard-coded secrets and outdated dependencies, we measure classical detection accuracy (pr…
-
Finding Fixed Vulnerabilities with Off-the-Shelf Static AnalysisT. Dunlap, Seaver Thorn, W. Enck et al. · 2023 · European Symposium on Security and Privacy · cité 17 fois · Pas encore résumé
Finding Fixed Vulnerabilities with Off-the-Shelf Static Analysis utilise les alertes SAST pour repérer des correctifs de vulnérabilités, tandis que cette étude examine leur correspondance avec les lignes corrigées.
Résumé scientifique
Software depends on upstream projects that regularly fix vulnerabilities, but the documentation of those vulnerabilities is often unreliable or unavailable. Automating the collection of existing vulnerability fixes is essential for downstream projects to reliably update their dependencies due to the sheer number of dependencies in modern software. Prior efforts rely solely on incomplete databases or imprecise or inaccurate statistical analysis of upstream repositories. In this paper, we introduce Differential Alert Analysis (DAA) to discover vulnerability fixes in software projects. In contrast to statistical analysis, DAA leverages static analysis security testing (SAST) tools, which reason…
-
Persistent Human Feedback, LLMs, and Static Analyzers for Secure Code Generation and Vulnerability DetectionEhsan Firouzi, Mohammad Ghafari · 2026 · IEEE International Conference on Software Analysis, Evolution, and Reengineering · cité 8 fois · Pas encore résumé
Persistent Human Feedback, LLMs, and Static Analyzers évalue CodeQL et Semgrep sur du code généré par des modèles, avec une vérité terrain validée par des humains.
Résumé scientifique
Existing literature heavily relies on static analysis tools to evaluate LLMs for secure code generation and vulnerability detection. We reviewed 1,080 LLM-generated code samples, built a human-validated ground-truth, and compared the outputs of two widely used static security tools, CodeQL and Semgrep, against this corpus. While 61% of the samples were genuinely secure, Semgrep and CodeQL classified 60% and 80% as secure, respectively. Despite the apparent agreement in aggregate statistics, per-sample analysis reveals substantial discrepancies: only 65% of Semgrep's and 61% of CodeQL's reports correctly matched the ground truth. These results question the reliability of static analysis tools…
-
SmartCode AI: A Hybrid Static Analysis and Ensemble Transformer Framework for Secure Code Review and Automated RemediationN. Maheswari, A. K. Prabhakaran, R. Vignesh et al. · 2026 · International Congress on Information and Communication Technology · Pas encore résumé
SmartCode AI associe analyse statique et modèles Transformer pour la revue et la correction de code, plutôt que d’étudier la stabilité temporelle des détecteurs.
Résumé scientifique
Application security vulnerabilities continue to dominate the threat landscape, contributing to nearly seventy percent of reported security breaches. The growing volume of software applications, combined with increasing architectural complexity, has significantly expanded the attack surface. Traditional Static Application Security Testing (SAST) tools remain widely adopted; however, existing approaches suffer from critical limitations. High false positive rates, often ranging between sixty and eighty percent, create alert fatigue and reduce developer trust. Pattern-based detection mechanisms lack contextual understanding of business logic, leading to misclassification of safe code as vulnera…
Les travaux antérieurs sur lesquels il s’appuie
-
Comparison and Evaluation on Static Application Security Testing (SAST) Tools for JavaKai-Xuan Li, Sen Chen, Ling-Ling Fan et al. · 2023 · ESEC/SIGSOFT FSE · cité 86 fois · Pas encore résumé
Comparison and Evaluation on Static Application Security Testing compare des outils SAST sur Java selon leur efficacité, leur cohérence et leurs performances, et est cité par cette étude.
Résumé scientifique
Static application security testing (SAST) takes a significant role in the software development life cycle (SDLC). However, it is challenging to comprehensively evaluate the effectiveness of SAST tools to determine which is the better one for detecting vulnerabilities. In this paper, based on well-defined criteria, we first selected seven free or open-source SAST tools from 161 existing tools for further evaluation. Owing to the synthetic and newly-constructed real-world benchmarks, we evaluated and compared these SAST tools from different and comprehensive perspectives such as effectiveness, consistency, and performance. While SAST tools perform well on synthetic benchmarks, our results ind…
-
How Many of All Bugs Do We Find? A Study of Static Bug DetectorsA. Habib, Michael Pradel · 2018 · International Conference on Automated Software Engineering · cité 130 fois · Pas encore résumé
How Many of All Bugs Do We Find? mesure la détection de bogues réels par des analyseurs statiques sur des projets Java et est cité par cette étude.
Résumé scientifique
Static bug detectors are becoming increasingly popular and are widely used by professional software developers. While most work on bug detectors focuses on whether they find bugs at all, and on how many false positives they report in addition to legitimate warnings, the inverse question is often neglected: How many of all real-world bugs do static bug detectors find? This paper addresses this question by studying the results of applying three widely used static bug detectors to an extended version of the Defects4J dataset that consists of 15 Java projects with 594 known bugs. To decide which of these bugs the tools detect, we use a novel methodology that combines an automatic analysis of war…
-
An empirical study of security warnings from static application security testing toolsBushra Aloraini, M. Nagappan, D. Germán et al. · 2019 · Journal of Systems and Software · cité 66 fois · Pas encore résumé
An empirical study of security warnings analyse les types et l’évolution des alertes SAST, et est cité par cette étude sur les variations des résultats.
Résumé scientifique
Abstract The Open Web Application Security Project (OWASP) defines Static Application Security Testing (SAST) tools as those that can help find security vulnerabilities in the source code or compiled code of software. Such tools detect and classify the vulnerability warnings into one of many types (e.g., input validation and representation). It is well known that these tools produce high numbers of false positive warnings. However, what is not known is if specific types of warnings have a higher predisposition to be false positives or not. Therefore, our goal is to investigate the different types of SAST-produced warnings and their evolution over time to determine if one type of warning is m…
-
Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source SoftwareM. Beller, R. Bholanath, Shane McIntosh et al. · 2016 · IEEE International Conference on Software Analysis, Evolution, and Reengineering · cité 234 fois · Pas encore résumé
Analyzing the State of Static Analysis évalue l’analyse statique à grande échelle dans les logiciels open source et est cité par cette étude.
-
Do Developers Use Static Application Security Testing (SAST) Tools Straight Out of the Box? A large-scale Empirical StudyGareth Bennett, Tracy Hall, Steve Counsell et al. · 2024 · International Symposium on Empirical Software Engineering and Measurement · cité 13 fois · Pas encore résumé
Do Developers Use Static Application Security Testing Tools Straight Out of the Box? étudie les pratiques de configuration SAST de développeurs, un facteur de contexte pour les résultats des analyseurs.
Résumé scientifique
Static application Security Testing (SAST) tools are an established means of detecting vulnerabilities early in development. Previous studies have reported low detection rates from SAST tools and recommend either combining SAST tools or configuring rule sets to detect more vulnerabilities. However, while previous work suggests that developers rarely combine or configure any of the Automatic Static Analysis Tools (ASATs) they use, it is currently unclear whether SAST tools are used directly “out of the box”. To understand how developers use SAST tools, we performed a large-scale survey involving 1,263 developers. We pre-screened developers to establish their SAST use and found that only 20% (…
Aller plus loin
Comment ces articles sont-ils trouvés ?
Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.
Mis à jour le 9 novembre 2026.