Google fixed 1,072 security bugs in two Chrome releases, more than the previous 23 combined. The post announcing it publishes no regression rate.
Not a low number. No number.
The Chrome Security team’s July 30 post describes the pipeline openly. A fixing agent returns candidate patches, a critic agent picks one, test-writing agents cover it before a developer reviews.
What went right is quantified: LLMs “generating candidate fixes for most vulnerabilities”, and an automated triage step Google estimates saves hundreds of developer hours a month.
What went wrong is not quantified at all. No revert count, no regression count, no rate at which the critic rejected the fixer’s work.
My take: Speed and throughput get measured everywhere. Whether the fix held is the one step with no published number, and it was supposed to be the verification.
Then the bug turns out not to exist
On the same day, JFrog published an analysis of six SQLite advisories from a new GitHub repository carrying 50+ CVEs it believes are LLM-generated.
NVD flagged them critical. CISA’s ADP agreed. Red Hat scored CVE-2026-51302 at 10.0 before downgrading it to 7.6.
For CVE-2026-51303, the diff between SQLite 3.51.2 and 3.51.3 shows “absolutely no changes to src/expr.c”. The patch the advisory described had never existed.
JFrog’s structural finding: “no step in today’s system actually requires a proof-of-concept or bug reproduction.”
Point an autofix agent at that chain and it is working on a bug that was never there. By JFrog’s own reading, no step downstream is set up to catch that.
When the scanner becomes the target
If an agent can iterate until the alert closes, the scanner has stopped verifying the fix and become the thing the fixer optimizes against.
“Alert closed” then means the agent reached a state this analyzer does not flag. That is a much weaker claim than the one most people read into it.
I sat on the vendor side of enough proof-of-concepts to know what a green scanner meant in that room. It was the acceptance criterion, and later a line on the renewal slide.
Nobody in those meetings asked what happened to the fix afterward. I did not ask either.
To be clear on the limit: this is a claim about incentives, not evidence that Google shipped defective fixes.
What the auditable numbers say
CVE-Bench (NAACL 2025) built a repair environment around 509 real CVEs from 120 open-source repositories: “the SWE-agent can only repair 21% of vulnerabilities at its best.”
That figure is 15 months old and SWE-agent is a general coding agent, not a security product. Both are fair objections.
Endor Labs’ Agent Security League is current, extending Carnegie Mellon’s SusVibes across 200 tasks from 108 Python projects and 77 CWE classes.
Across every agent and model combination, the highest functional score is 84.9% and the highest secure score is 29%.
Not a leaderboard — CVE-Bench measures repair, Endor measures generation, and neither evaluates Chrome. But where the methodology is published and checkable, correctness lands far below the speed marketing.
GitHub is the most candid party here. Its autofix launch post claims suggestions that “remediate more than two-thirds of found vulnerabilities with little or no editing”.
Its responsible-use documentation then lists syntax errors, location errors and semantic errors, adding: “The system has no understanding of the programmer’s intent.”
One question for you
Has anyone published a revert rate for AI-generated security fixes — vendor, internal, academic, anything? I could not find one, and I would rather be wrong about that.
Send it over and I will run it next issue with the source.
Sources
- Stronger with every update — Chrome Security Team, Google, July 30, 2026. Vendor-authored.
- SQLite Critical CVEs or LLM Slop? — JFrog Security Research, July 30, 2026. Vendor-authored.
- CVE-Bench — Wang, Liu & Xiao, NAACL 2025, pages 4207–4224. Peer-reviewed.
- Agent Security League — Endor Labs, open benchmark extending SusVibes (Carnegie Mellon). Vendor-authored, methodology published.
- Found means fixed and Responsible use of AI security features — GitHub. Vendor-authored.