The CandyShop benchmark records output from 12 tools in five categories — SAST, DAST, SCA, container scanning, and IaC — across 6 intentionally vulnerable applications (OWASP Juice Shop, Broken Crystals, Altoro Mutual, vulnpy, DVWA, and WebGoat).
The archived table contains 10,047 finding records. Raw outputs, exact versions and final triage files are not preserved, so the run cannot be independently reproduced from this repository. The prior precision, recall and F1 calculations are withdrawn as accuracy evidence; the finding counts remain available below.
Key Findings#
1. Container-image findings dominate this snapshot#
DVWA’s PHP/Apache image produced 3,672 container findings (Grype + Trivy combined). Juice Shop’s Node.js image: 271.
These are different applications and dependency sets, so this comparison does not isolate base-image choice as the sole cause. It does show why operating-system packages deserve attention when reviewing a container scan.
2. More findings does not mean better detection#
Grype reported 5,046 findings across all 6 targets — the highest count from any tool. The vast majority came from base image OS packages, not application-level flaws. npm audit found 99 findings total, but 9 were critical and 46 were high. Look at severity distribution, not totals.
3. No single scanner catches everything#
Scanners operate on different inputs and can leave different gaps. The saved data cannot quantify how much each tool missed; choose complementary tools by coverage needs and verify their findings independently.
4. Container scanners and SCA tools inspect different inputs#
Trivy and Grype inspect container packages, while npm audit and pip-audit focus on application dependencies. On Juice Shop, Trivy recorded 135 findings and npm audit 56. Counts alone do not reveal overlap between those findings.
5. DAST coverage depends on authentication#
ZAP consistently found 5-20 issues per target, mostly medium or lower severity. Without login credentials, ZAP only tests what an anonymous visitor can reach. The gap between 13 findings on Juice Shop and 20 on DVWA says more about how deep the login wall sits than about actual vulnerability counts.
6. IaC scanning adds configuration checks#
Checkov flagged Dockerfile misconfigurations across 3 targets (Juice Shop, vulnpy, DVWA). Running containers as root, skipping health checks — these aren’t “vulnerabilities” in the traditional sense, but they’re real security problems that SAST, SCA, and DAST tools all ignore.
What Do the Archived Scores Show?#
The previous calculation labeled 654 cross-tool matches as true positives. Matching reports are a review signal, not independent ground truth. The following table preserves those old calculations for transparency; it must not be used to rank detection quality.
Precision was forced to 1.000 by leaving unconfirmed findings out of the false-positive denominator. Without independent labels and the original triage records, neither the reported recall nor F1 can be validated.
Withdrawn consensus calculations — not validated scanner accuracy
| Tool | Avg F1 | Precision | Recall | TP | FP | CWEs |
|---|---|---|---|---|---|---|
| Trivy | 0.783 | 1.000 | 0.662 | 309 | 0 | 25 |
| FindSecBugs | 0.707 | 1.000 | 0.571 | 62 | 0 | 7 |
| OpenGrep | 0.645 | 1.000 | 0.490 | 109 | 0 | 13 |
| Bandit | 0.625 | 1.000 | 0.455 | 10 | 0 | 4 |
| Grype | 0.528 | 1.000 | 0.382 | 92 | 0 | 5 |
| Dependency-Check | 0.400 | 1.000 | 0.263 | 27 | 0 | 10 |
| npm audit | 0.394 | 1.000 | 0.246 | 19 | 0 | 10 |
| OWASP ZAP | 0.260 | 1.000 | 0.164 | 20 | 0 | 6 |
| Nuclei | 0.090 | 1.000 | 0.048 | 3 | 0 | 0 |
| NodeJsScan | 0.077 | 1.000 | 0.040 | 3 | 0 | 1 |
How Do Different Scanner Categories Compare?#
The charts now show the recorded finding counts from the heatmap below. They compare output volume within each category. For accuracy testing, the OWASP Benchmark illustrates the need for labeled expected results; agreement between scanners is not a substitute.
How to read these charts:
- Finding-count chart (above) shows all 12 tools using their recorded totals, not withdrawn F1 values.
- Category charts use the same counts for tools with similar input scopes. A longer bar means more output records, not better detection.
- pip-audit and Checkov remain in the overall chart and heatmap. Their records should not be discarded because they lack a matching report from another category.
SAST Tools#
FindSecBugs recorded 192 findings across 2 Java targets. OpenGrep recorded 456 across all 6 targets; the different language and target coverage matters when comparing volume.
Bandit recorded 87 findings on vulnpy; NodeJsScan recorded 14 across the two JavaScript targets. The archive does not establish their false-positive rates.
Container Scanners#
Trivy recorded 3,854 findings and Grype 5,046. Both inspect packages inside container images; their totals do not tell us which scanner was more accurate.
SCA Tools#
Dependency-Check recorded 137 findings across its applicable targets, npm audit 99 across the JavaScript targets, and pip-audit 14 on vulnpy. Their ecosystems differ, so a total is not an interchangeable measure of coverage.
DAST Tools#
ZAP recorded 83 findings and Nuclei 57. Crawl configuration, templates and authentication affect DAST coverage; the archived totals do not establish a winner.
IaC Scanning#
Checkov is the only IaC tool in the benchmark. It flagged Dockerfile misconfigurations in 3 targets (Juice Shop, vulnpy, DVWA) — running containers as root, missing health checks, using latest tags.
Checkov recorded 8 findings across those targets. Its IaC scope differs from source, dependency and runtime scanning; the overall chart includes its output without treating cross-tool agreement as a validity test.
How Many Findings Did Each Tool Record?#
The heatmap below preserves total finding records per tool and target. Darker red means more records, not more unique or confirmed vulnerabilities. The expandable notes reproduce observations from the archived page; their severity and root-cause details cannot be independently checked without the raw reports.
| Tool | Juice Shop | Broken Crystals | Altoro Mutual | vulnpy | DVWA | WebGoat | Total |
|---|---|---|---|---|---|---|---|
| Grype | 136 | 2,111 | 62 | 144 | 2,097 | 496 | 5,046 |
| Trivy | 135 | 1,555 | 50 | 136 | 1,575 | 403 | 3,854 |
| OpenGrep | 70 | 42 | 46 | 12 | 100 | 186 | 456 |
| FindSecBugs | — | — | 54 | — | — | 138 | 192 |
| Dep-Check | 0 | 66 | 23 | 0 | 1 | 47 | 137 |
| npm audit | 56 | 43 | — | — | — | — | 99 |
| Bandit | — | — | — | 87 | — | — | 87 |
| ZAP | 13 | 5 | 17 | 14 | 20 | 14 | 83 |
| Nuclei | 14 | 12 | 12 | 5 | 10 | 4 | 57 |
| Checkov | 3 | 0 | 0 | 2 | 3 | 0 | 8 |
| pip-audit | — | — | — | 14 | — | — | 14 |
| NodeJsScan | 1 | 13 | — | — | — | — | 14 |
— = tool not applicable to this target's language/framework. Color scale: 0–10 11–50 51–200 201–500 501–1000 1000+
OWASP Juice Shop — 428 total findings across 8 tools
- OpenGrep found 70 issues with 38 at high severity — the only SAST tool to flag high-severity vulnerabilities on Juice Shop.
- Grype and Trivy reported nearly identical totals (136 vs 135) with similar severity distributions, which is reassuring — the two container scanners largely agree.
- npm audit found 7 critical and 31 high-severity dependency vulnerabilities.
Broken Crystals — 3,847 total findings across 8 tools
- Grype produced 2,111 findings — the heaviest base image in the benchmark, with 30 critical and 511 high-severity issues.
- Trivy hit 1,555 findings. The bloated base image explains the jump from Juice Shop’s 135.
- OpenGrep found 42 issues (26 high severity), while NodeJsScan caught 13 including 10 high-severity findings (hardcoded credentials and eval injection).
- Dependency-Check found 66 issues versus zero on Juice Shop — richer dependency trees give it more to work with.
- ZAP found only 5 issues despite 20+ vulnerability types in the target. Without authentication, DAST tools just can’t reach enough of the attack surface.
Altoro Mutual — 264 total findings across 7 tools
- FindSecBugs led with 54 findings, including 10 SQL injection, 3 path traversal, and 1 XXE. This is the only target where a Java-specific SAST tool outperformed container scanners.
- OpenGrep found 46 issues (13 high, 33 medium), picking up source-level patterns that FindSecBugs missed.
- Trivy reported 50 container findings including 5 critical CVEs in the Java runtime layer.
- ZAP recorded 17 findings here, compared with 20 on DVWA. The totals alone do not explain differences in crawl coverage.
vulnpy — 414 total findings across 8 tools
- Bandit is the only Python-specific SAST scanner in the benchmark. It found 87 informational issues — mostly
eval(),exec(), andsubprocessusage. - Trivy found 136 container vulnerabilities, 107 of them low severity. The Python base image has a moderate vulnerability surface.
- pip-audit found 14 medium-severity issues — a clean, focused set compared to the container scanning noise.
- Interesting coincidence: ZAP and pip-audit both returned 14 findings, from completely different angles (runtime vs dependency analysis).
DVWA — 3,806 total findings across 7 tools
- One of the highest-volume targets in the archived notes. Grype alone reported 2,097 findings and Trivy added 1,575. The PHP/Apache base image is a CVE magnet — 327 critical findings from Grype.
- Nuclei found a critical-severity issue here — the only critical from any DAST tool across all 6 targets. An exposed admin panel / known vulnerable endpoint.
- Dependency-Check found only 1 medium-severity issue. PHP/Composer gets much less SCA coverage than npm or Maven.
WebGoat — 1,288 total findings across 7 tools
- OpenGrep found 186 issues — the highest SAST count in the benchmark. The Java/Spring codebase triggered 44 high-severity and 142 medium-severity findings.
- FindSecBugs found 138 issues, including 14 SQL injection, 19 path traversal, and 14 Spring CSRF findings. Its bytecode analysis catches patterns that source-level scanners miss.
- Grype (496) and Trivy (403) had similar severity distributions here too — container scanners agree consistently.
- Dependency-Check had its best result here with 47 issues. Java/Maven is the ecosystem it handles best.
What Tools and Targets Are in the Benchmark?#
Tools Tested#
The CandyShop benchmark tests 12 open-source tools across five categories: SAST (OpenGrep, NodeJsScan, Bandit, FindSecBugs), DAST (OWASP ZAP, Nuclei), SCA (npm audit, pip-audit, OWASP Dependency-Check), container scanning (Trivy, Grype), and IaC (Checkov). All use open-source licenses (Apache 2.0, MIT, LGPL, GPL) — no commercial scanners, no vendor agreements needed.
| Category | Tools Tested |
|---|---|
| SAST | OpenGrep, NodeJsScan, Bandit, FindSecBugs |
| DAST | OWASP ZAP, Nuclei |
| SCA | npm audit, pip-audit, OWASP Dependency-Check |
| Container | Trivy, Grype |
| IaC | Checkov |
Test Targets#
6 intentionally vulnerable applications spanning Node.js, Java, Python, and PHP:
| Target | Stack | Vulnerabilities | Notes |
|---|---|---|---|
| Juice Shop | Node.js/Express/Angular | 100+ challenges | Most widely used vulnerable app |
| Broken Crystals | Node.js/TypeScript | 40+ types | JWT flaws, XXE, business logic |
| Altoro Mutual | J2EE | Classic web vulns | SQL injection, XSS, path traversal |
| vulnpy | Python/Flask | 13 categories | Python-specific scanner testing |
| DVWA | PHP/MariaDB | Adjustable levels | Classic training ground |
| WebGoat | Java/Spring | Guided lessons | OWASP teaching application |
The original notes describe Docker Compose and default scanner settings. The saved repository lacks a complete version/configuration manifest, so that setup cannot be verified or reproduced exactly.
How Is the Benchmark Methodology Designed?#
Environment Setup#
The original setup used Docker Compose. The surviving Compose file does not include all six target services or all referenced build contexts, and the scanner versions and commands were not preserved.
Tool Selection Criteria#
The original selection notes listed three intended requirements; the archive does not preserve a version-by-version checklist:
- Open-source license (Apache 2.0, MIT, LGPL, GPL, or similar). No commercial tools, no freemium tiers, no “community editions” with half the features stripped out.
- Active maintenance — last commit within the past 12 months.
- CLI-driven — can run headless in a CI pipeline without a GUI.
How Is Ground Truth Established?#
The earlier multi-tool consensus model treated matching CWE/location reports from two tools as true positives. That assumption does not establish ground truth: tools can agree on a false positive, and category-specific findings may have no second scanner to confirm them.
Single-tool findings can be valid. The previously reported 152-entry reference set and its final triage decisions are not preserved, so the old counts cannot be independently validated.
Agreement can help prioritize triage, but it cannot prove precision or supply the missing false negatives. The old model could both exclude real issues and accept shared false positives.
How Is F-Measure Calculated?#
F-measure (F1 score) is the harmonic mean of precision and recall. The standard formulas are:
- Precision = TP / (TP + FP) — how many of the tool’s confirmed findings are real
- Recall = TP / (TP + FN) — how many of the known ground-truth issues the tool detected
- F1 Score = 2 * (Precision * Recall) / (Precision + Recall)
The old calculation excluded unconfirmed findings from FP and therefore reported precision of 1.000. That is a property of the calculation, not evidence that every finding was correct.
The preserved score table is historical only. A new accuracy comparison would need versioned targets, independent expected-result labels, complete scanner output and adjudication of every relevant finding.
What these numbers can’t tell you#
The archived counts cannot establish precision, recall, a false-positive rate, or a lower bound on coverage. Their useful scope is comparing recorded output volume across tools and targets.
The snapshot covers six deliberately vulnerable applications. Its finding counts do not predict what a current tool will find in your production stack.
Related guides: