Q003 — CI Tooling in Major OSS Projects — Assessment¶
Contents¶
Direct evidence was found primarily for the Linux kernel and partially for PostgreSQL. The kernel uses four bespoke static analysis tools (checkpatch.pl, Sparse, Smatch, Coccinelle) integrated with the kernel Makefile rather than standard SAST tools. False positives and lack of kernel-specific semantics are cited by kernel maintainer Shuah Khan as primary reasons. PostgreSQL uses cfbot for bespoke patch-review CI. Evidence for Node.js, Kubernetes, and CPython was not found in this investigation. The pattern of building bespoke tooling is well-documented for the kernel but extrapolated to other projects.
Evidence Synthesis¶
Evidence quality: Medium — Strong evidence for the Linux kernel's static analysis tooling from a New Stack interview with kernel maintainer Shuah Khan. Moderate evidence for GitHub Actions dependency tooling from the Mons/Radboud paper. Weak evidence for PostgreSQL (cfbot wiki page only). No direct evidence found for Node.js, Kubernetes, or CPython CI tooling choices within the fetched sources.
Source agreement: Medium — Sources agree on the general pattern that large projects use specialized/bespoke tooling, but coverage is uneven. The kernel evidence is detailed and authoritative. The PostgreSQL evidence is limited. Three of five projects (Node.js, Kubernetes, CPython) have no direct evidence from this investigation.
Independence: The available sources are independent — The New Stack interview with a kernel maintainer, the academic paper on GHA workflows, and the PostgreSQL wiki are unrelated evidence streams.
Probability Assessment¶
Confidence: Medium
Evidence Gaps¶
Expected but not found: - No evidence about Node.js CI tooling was found in the fetched sources. - No evidence about Kubernetes CI tooling (Prow, Tide) and security scanning was found. - No evidence about CPython CI tooling and quality enforcement was found. - No documented evaluation-and-rejection of standard SAST/SCA tools by any of the five projects was found. - No systematic comparison of false-positive rates between bespoke and standard tools on large codebases was found.
Unanswered questions: - Does Kubernetes use CodeQL or any SAST scanner in its Prow-based CI? - Does CPython use any security scanning tools in its GitHub Actions CI? - Does Node.js use any security scanning beyond Dependabot? - Have any of these projects formally evaluated and rejected specific SAST/SCA tools with documented rationale?
Impact on confidence: The gaps for 3 of 5 named projects significantly reduce confidence. The answer is primarily driven by the Linux kernel case, which may not be representative. A complete answer would require examining each project's CI configuration and development mailing lists, which was not accomplished within the search scope.