Run Overview¶
Contents¶
- Axioms
- Claims
- C001 — OSS CI Adoption Rate Around 40% — Likely to Very likely (75-90%)
- C002 — OpenSSF Scorecard Average of 5.4 Out of 10 — Unlikely as stated (20-45%)
- C003 — OSSRA Report: 87% Vulnerable, 581 Avg, 107% Increase — Almost certain (95-99%) that the OSSRA report contains these figures
- C004 — 95% of Vulnerable Components Have Available Fix — Almost certain (95-99%)
- Queries
- Collection Analysis
Axioms¶
A001¶
This research is conducted in the context of understanding open source CI/CD practices to evaluate whether a comprehensive, zero-configuration auditing stack (enforcing SAST, SCA, container scanning, strict type checking, SBOM generation, and 100% code coverage as CI gates) represents meaningful differentiation in the open source ecosystem.
Claims¶
C001 — OSS CI Adoption Rate Around 40% — Likely to Very likely (75-90%)¶
Claim: Only approximately 40% of open source projects use continuous integration at all.
Verdict: The claim that approximately 40% of open source projects use CI is well-supported by multiple independent empirical studies. The exact figure varies by study and sample composition (40% in 2016, 43.9% in 2022, 50%+ in npm-specific repos), but the approximate 40% figure serves as a reasonable baseline. The claim masks significant stratification: CI adoption is much higher among actively maintained, popular projects.
| Hypothesis | Status |
|---|---|
| H1: ** | — |
| H2: ** | — |
| H3: ** | — |
| H4: ** | — |
Sources: 13 · Searches: 5
C002 — OpenSSF Scorecard Average of 5.4 Out of 10 — Unlikely as stated (20-45%)¶
Claim: The average OpenSSF Scorecard score across the top one million critical open source projects is 5.4 out of 10.
Verdict: The claim conflates two separate facts: OpenSSF scans 1 million critical projects weekly (confirmed), and the average Scorecard score is 5.4 (confirmed for Chainguard's analysis of 1,511 Wolfi packages). No source combines these into the claim as stated. The 5.4 figure is plausible for the broader ecosystem (Chainguard notes it is 'typical') but is not confirmed for the 1M critical project population.
| Hypothesis | Status |
|---|---|
| H1: ** | — |
| H2: ** | — |
| H3: ** | — |
Sources: 13 · Searches: 4
C003 — OSSRA Report: 87% Vulnerable, 581 Avg, 107% Increase — Almost certain (95-99%) that the OSSRA report contains these figures¶
Claim: 87% of audited codebases contain at least one known open source vulnerability, with an average of 581 vulnerabilities per codebase representing a 107% year-over-year increase.
Verdict: All three figures (87% vulnerability prevalence, 581 average vulnerabilities per codebase, 107% YoY increase) are confirmed from the Black Duck 2026 OSSRA report. However, the sample consists of 947 commercial codebases undergoing M&A audits, not open source projects. This commercial audit sample systematically overrepresents codebases with deferred maintenance and neglected dependency management, making the figures potentially misleading when used to characterize the open source ecosystem generally.
| Hypothesis | Status |
|---|---|
| H1: ** | — |
| H2: ** | — |
| H3: ** | — |
Sources: 13 · Searches: 4
C004 — 95% of Vulnerable Components Have Available Fix — Almost certain (95-99%)¶
Claim: 95% of vulnerable open source components consumed by downstream projects have a known fix available that has not been applied.
Verdict: The 95% figure (precisely 94.9%) is confirmed from three Sonatype publications as part of their 2024 State of the Software Supply Chain report. The data covers Maven Central, npm, PyPI, and NuGet and is longitudinally consistent with prior years (96% in 2022-2023). The figure means that within a year, a non-vulnerable version was available for nearly all vulnerable components that were downloaded — making these vulnerable downloads 'avoidable.'
| Hypothesis | Status |
|---|---|
| H1: ** | — |
| H2: ** | — |
| H3: ** | — |
Sources: 13 · Searches: 4
Queries¶
Q001 — Comprehensive CI Gate Adoption Rates in OSS — Low¶
Query: What fraction of open source projects on GitHub enforce comprehensive CI gate tooling — specifically static analysis (SAST), dependency/vulnerability scanning (SCA), container image scanning, strict type checking, and code coverage thresholds — as required checks on every commit or pull request? How does adoption vary by project size, programming language ecosystem, and whether the project is backed by a foundation or company versus community-maintained?
Answer: The fraction of OSS projects enforcing comprehensive CI gate tooling is unmeasured but almost certainly very small (likely single-digit percentages). Dependency update tools show the highest adoption (Dependabot 69.2%, Renovate 21.0% in GHA workflows), but these primarily update Action versions, not scan for vulnerabilities. SAST adoption is poorly quantified and appears low. Container scanning, type checking enforcement, and coverage threshold enforcement are essentially unquantified for the OSS ecosystem. No study measures co-adoption of all five tool categories simultaneously.
Confidence: Low · Sources: 13 · Searches: 6
Q002 — Barriers to OSS Security Scanning Adoption — Medium¶
Query: What are the primary barriers to adoption of comprehensive security scanning pipelines in open source projects, and what evidence exists for 'security fatigue' — the phenomenon where initial post-incident surges in security tooling adoption (e.g., after Log4Shell) reverse over time? What role do setup complexity, false-positive rates, and maintenance burden play in driving abandonment?
Answer: The primary barriers are: (1) lack of awareness of available security tools and features; (2) perception that security tooling is not necessary; (3) extreme false-positive rates of 60-91% out of the box creating alert fatigue; (4) supply chain mistrust and lack of automation. Direct empirical evidence for temporal 'security fatigue' (post-incident surges followed by reversal) was not found. Log4j data shows persistent vulnerable consumption declining slowly (from 30-35% to 13% over three years), consistent with gradual attention decay but not a clear surge-and-reversal cycle.
Confidence: Medium · Sources: 13 · Searches: 5
Q003 — CI Tooling in Major OSS Projects — Medium¶
Query: Among well-resourced open source projects (Linux kernel, PostgreSQL, Node.js, Kubernetes, Python CPython), what CI and quality enforcement tooling do they actually run, and why do they build bespoke tooling rather than adopting standard commercial or open-source SAST/SCA scanners like CodeQL, Semgrep, and Trivy? Is there evidence that these tools are considered inadequate, too noisy, or inapplicable to large-scale projects?
Answer: Direct evidence was found primarily for the Linux kernel and partially for PostgreSQL. The kernel uses four bespoke static analysis tools (checkpatch.pl, Sparse, Smatch, Coccinelle) integrated with the kernel Makefile rather than standard SAST tools. False positives and lack of kernel-specific semantics are cited by kernel maintainer Shuah Khan as primary reasons. PostgreSQL uses cfbot for bespoke patch-review CI. Evidence for Node.js, Kubernetes, and CPython was not found in this investigation. The pattern of building bespoke tooling is well-documented for the kernel but extrapolated to other projects.
Confidence: Medium · Sources: 13 · Searches: 6
Collection Analysis¶
Collection Statistics¶
| Metric | Value |
|---|---|
| Items investigated | 8 |
| Sources scored | 13 |
| Searches executed | 17 |