Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?
How to prepare an accurate paper press kit? Illustrated guide
Junhua Tang, Zhengsong Zhang. Haipeng Cai · arXiv: 2611.03742v1 · Summary page
Why cover this story this week?
Security scanners can improve their scores without improving discrimination
Does a security scanner configured today remain useful as software changes? A study of 20,000 vulnerable and fixed C/C++ cases compared releases of CodeQL and Infer alongside a Semgrep configuration. The researchers examined yearly populations, matched projects across years, and traced selected performance drops through repair edits and analysis logs. Newer CodeQL releases achieved higher F1 scores, but recall and false-positive rates rose together; a separate measure of discrimination remained near chance.
The investigation exposes a measurement problem as much as a software problem. Many examined alerts did not touch the lines changed by a repair, while Semgrep reported successful runs even when parser or timeout diagnostics were present. A tool can appear to improve by issuing more warnings, or appear to deteriorate because its analysis coverage changes. For anyone procuring or evaluating security software, the lesson is to ask what an alert actually demonstrates—not merely whether a headline score went up.
Editorial context written by AI from abstracts, not independent scientific validation. The kit and its verified quotes rely on the source paper, not on this letter.
Talk about a paper in public without overstating it: yours, one you are asked to comment on, or one you write about as a journalist.
- Your paper: a draft release for your press office, the questions journalists will ask with what the paper lets you answer, the likely misreadings, a lay summary and a short thread.
- Asked to comment: what the paper shows and does not, its strengths and weaknesses with their passages, and what would settle its main claim.
- Writing about it as a journalist: reporting angles, the wording to check before publication, and the questions to ask the authors.
- For everyone: findings with their passages checked word for word, the novelty set against the closest earlier papers, a hype check, figures with their credit, and researchers for an outside view who have not signed with the authors.