Run Overview¶
Pipeline: ❌ 4 issues (1 sources dropped) — details
Contents¶
- Axioms
- Claims
- C001 — CI Adoption Rate: 40% Figure Is Outdated — Likely correct historically (55-80%), very likely outdated (80-95%)
- C002 — OpenSSF Scorecard 5.4 Average: Wrong Population — Unlikely as stated (20-45%)
- C003 — OSSRA Vulnerability Figures: Accurate but Contextual — Almost certainly accurate as quotation (95-99%), contextual factors very likely influential (80-95%)
- C004 — 95% Fixable Vulnerabilities: Accurate but Misleading — Almost certainly accurate (95-99%) but very likely misleading (80-95%)
- Queries
- Collection Analysis
- Pipeline Notes
Axioms¶
A001¶
This research is conducted in the context of understanding open source CI/CD practices to evaluate whether a comprehensive, zero-configuration auditing stack (enforcing SAST, SCA, container scanning, strict type checking, SBOM generation, and 100% code coverage as CI gates) represents meaningful differentiation in the open source ecosystem.
Claims¶
C001 — CI Adoption Rate: 40% Figure Is Outdated — Likely correct historically (55-80%), very likely outdated (80-95%)¶
Claim: Only approximately 40% of open source projects use continuous integration at all.
Verdict: The 40% CI adoption figure is likely correct as a historical data point from the seminal 2016 Hilton et al. study, but is very likely outdated as a description of current adoption. GitHub Actions alone matched the entire 40% rate by January 2022. The figure is also sensitive to population definition — active projects show much higher CI adoption than the full repository population including abandoned and hobby projects. The researcher should not present 40% as a current figure.
| Hypothesis | Status |
|---|---|
| H1: ** | — |
| H2: ** | — |
| H3: ** | — |
| H4: ** | — |
Sources: 16 · Searches: 5
C002 — OpenSSF Scorecard 5.4 Average: Wrong Population — Unlikely as stated (20-45%)¶
Claim: The average OpenSSF Scorecard score across the top one million critical open source projects is 5.4 out of 10.
Verdict: The claim misattributes the 5.4 average to 'the top one million critical open source projects' when it actually comes from a Chainguard analysis of approximately 1,511 Wolfi upstream repositories. The 5.4 figure may be a reasonable rough estimate of typical Scorecard scores across open source — Chainguard notes scores in the 4-6 range are 'typical' based on past research — but the specific population attribution is incorrect. The researcher should cite the Chainguard analysis directly or verify against the OpenSSF BigQuery dataset.
| Hypothesis | Status |
|---|---|
| H1: ** | — |
| H2: ** | — |
| H3: ** | — |
Sources: 16 · Searches: 4
C003 — OSSRA Vulnerability Figures: Accurate but Contextual — Almost certainly accurate as quotation (95-99%), contextual factors very likely influential (80-95%)¶
Claim: 87% of audited codebases contain at least one known open source vulnerability, with an average of 581 vulnerabilities per codebase representing a 107% year-over-year increase.
Verdict: The claim is almost certainly accurate as a direct quotation from the OSSRA 2026 report. However, the 107% increase is very likely influenced by measurement factors — codebase size grew 74% in files and 30% in components, the CVE database grew 263% over 5 years, and the Linux kernel CNA addition contributed 5,000+ new CVEs during the audit period. The OSSRA sample (947 codebases from M&A due diligence) is not representative of all codebases. Critical/high-severity vulnerability prevalence actually decreased slightly despite the total count increase. The researcher should present these as OSSRA audit findings with full contextual caveats, not as representative of all software.
| Hypothesis | Status |
|---|---|
| H1: ** | — |
| H2: ** | — |
| H3: ** | — |
| H4: ** | — |
Sources: 16 · Searches: 5
C004 — 95% Fixable Vulnerabilities: Accurate but Misleading — Almost certainly accurate (95-99%) but very likely misleading (80-95%)¶
Claim: 95% of vulnerable open source components consumed by downstream projects have a known fix available that has not been applied.
Verdict: The 95% figure is almost certainly accurate as a direct quotation from the Sonatype 2024 report (94.9% rounded). However, it is very likely misleading without context: 'fix available' means only that a patched version exists in the registry, not that applying it is trivial. Moderne's independent analysis shows only 30% of fixes are patch-level bumps — 50% require minor and 10% require major version updates, meaning 70% involve potentially breaking changes. Sonatype's own data shows mean remediation times growing from 25 days (2017) to 400+ days (2024). The researcher should present the 95% alongside this practical-barriers context.
| Hypothesis | Status |
|---|---|
| H1: ** | — |
| H2: ** | — |
| H3: ** | — |
Sources: 16 · Searches: 4
Queries¶
Q001 — Comprehensive CI Gate Adoption Is Near Zero — Medium¶
Query: What fraction of open source projects on GitHub enforce comprehensive CI gate tooling — specifically static analysis (SAST), dependency/vulnerability scanning (SCA), container image scanning, strict type checking, and code coverage thresholds — as required checks on every commit or pull request? How does adoption vary by project size, programming language ecosystem, and whether the project is backed by a foundation or company versus community-maintained?
Answer: The fraction of open source projects enforcing all five CI gate categories as required checks is negligibly small — likely well below 0.1% of all GitHub projects. Individual tool adoption is measurable but low: CodeQL at ~200K repos, Dependabot at ~846K repos. No data exists for container scanning, type checking, or coverage threshold enforcement as blocking checks in OSS CI.
Confidence: Medium · Sources: 16 · Searches: 6
Q002 — Barriers to Security Scanning: Complexity Over Fatigue — Medium¶
Query: What are the primary barriers to adoption of comprehensive security scanning pipelines in open source projects, and what evidence exists for 'security fatigue' — the phenomenon where initial post-incident surges in security tooling adoption (e.g., after Log4Shell) reverse over time? What role do setup complexity, false-positive rates, and maintenance burden play in driving abandonment?
Answer: The primary barriers are complexity/awareness, false positive rates, and maintenance burden. Evidence for 'security fatigue' as a temporal pattern is suggestive but not directly demonstrated. The pattern of persistent vulnerable consumption is better characterized as structural friction than episodic fatigue.
Confidence: Medium · Sources: 16 · Searches: 5
Q003 — Linux Kernel Uses Bespoke CI; Others Unstudied — Medium¶
Query: Among well-resourced open source projects (Linux kernel, PostgreSQL, Node.js, Kubernetes, Python CPython), what CI and quality enforcement tooling do they actually run, and why do they build bespoke tooling rather than adopting standard commercial or open-source SAST/SCA scanners like CodeQL, Semgrep, and Trivy? Is there evidence that these tools are considered inadequate, too noisy, or inapplicable to large-scale projects?
Answer: The Linux kernel uses an extensive suite of domain-specific tools rather than standard SAST/SCA. CNCF allows project-level tool choice without mandating scanners. Three of five target projects (PostgreSQL, Node.js, CPython) were not adequately covered and require separate investigation.
Confidence: Medium · Sources: 16 · Searches: 6
Collection Analysis¶
Collection Statistics¶
| Metric | Value |
|---|---|
| Items investigated | 8 |
| Sources scored | 16 |
Pipeline Notes¶
| Category | Count | Detail |
|---|---|---|
| Fetch failures (HTTP) | 1 | Fetch failed for https://dl.acm.org/doi/10.1145/3696630.3728699: 403 Client Erro |
| Missing step output | 3 | Expected output file self-audit.json not found in run directory (and 2 more) |
Fetch failures (HTTP)¶
https://dl.acm.org/doi/10.1145/3696630.3728699— Fetch failed for https://dl.acm.org/doi/10.1145/3696630.3728699: 403 Client Error: Forbidden for url: https://dl.acm.org/doi/10.1145/3696630.3728699
Missing step output¶
- — Expected output file self-audit.json not found in run directory
- — Expected output file reports.json not found in run directory
- — Expected output file reports.json not found in run directory