R0051/2026-03-31¶
Three queries investigating whether the fact-checking community has developed formal evidence evaluation frameworks, the status of the W3C Credibility Coalition's work, and whether academic literature has documented the absence of such frameworks as a gap.
Queries¶
Q001 — Epistemological Frameworks — High Confidence
Query: Has the fact-checking community developed any formal epistemological framework for evidence evaluation comparable to GRADE, IPCC, or ICD 203?
Answer: No formal epistemological framework comparable to GRADE, IPCC, or ICD 203 exists within the fact-checking community.
| Hypothesis | Status | Probability |
|---|---|---|
| H1: Formal frameworks exist | Eliminated | — |
| H2: Partial frameworks exist but none comparable | Supported | — |
| H3: No frameworks of any kind exist | Eliminated | — |
Confidence: High · Sources: 6 · Searches: 3
Q002 — W3C Credibility Coalition — High Confidence
Query: What is the current status of the W3C Credibility Coalition's CCIV and credibility signals work? Does it include hierarchical evidence quality scale, calibrated confidence language, structured bias assessment, or source reliability tiering?
Answer: Functionally dormant. CCIV is archival. Credibility Signals spec is an incomplete draft. Of four features queried, only rudimentary confidence calibration exists.
| Hypothesis | Status | Probability |
|---|---|---|
| H1: Actively maintained with features | Eliminated | — |
| H2: Dormant with limited features | Supported | — |
| H3: Fully abandoned | Eliminated | — |
Confidence: High · Sources: 5 · Searches: 3
Q003 — Documented Gap — High Confidence
Query: Has the academic literature identified and documented the absence of formal evidence evaluation frameworks in fact-checking as a gap?
Answer: Yes — multiple papers from 2013-2026 explicitly document the gap. No paper proposes filling it with a GRADE/IPCC/ICD 203-comparable framework.
| Hypothesis | Status | Probability |
|---|---|---|
| H1: Gap documented + solution proposed | Eliminated | — |
| H2: Gap documented, no GRADE-like solution | Supported | — |
| H3: Gap not documented | Eliminated | — |
Confidence: High · Sources: 5 · Searches: 3
Collection Analysis¶
Cross-Cutting Patterns¶
| Pattern | Queries Affected | Significance |
|---|---|---|
| Institutional infrastructure without methodological infrastructure | Q001, Q002 | Fact-checking has developed organizations, codes, and platforms but not formal evidence evaluation methods |
| 13-year documented gap without resolution | Q001, Q003 | The gap was first documented in 2013 (Uscinski & Butler) and remains unfilled in 2026 — sustained scholarly awareness without action |
| Diagnostic search absence as evidence | Q001, Q003 | Pairing "evidence quality/hierarchy" with "fact-checking" returned almost entirely medical results — the vocabulary has not crossed domains |
| Specification stall pattern | Q002 | Both CCIV (archival) and Credibility Signals (draft with TBDs) show the same trajectory — ambitious vocabulary projects that stalled before standardization |
| AI-era urgency without tools | Q001, Q003 | Generative AI deepens the need for formal evidence evaluation (Cazzamatta 2026) but no tools are emerging |
Collection Statistics¶
| Metric | Value |
|---|---|
| Queries investigated | 3 |
| High confidence | 3 (Q001, Q002, Q003) |
| H2 (partial/nuanced) supported | 3/3 — all queries resolved to the nuanced middle hypothesis |
Source Independence Assessment¶
The evidence base spans 16 unique sources across the three queries, drawn from multiple independent research groups:
- Theoretical/analytical: Vandenberghe (2025), Uscinski & Butler (2013), Cazzamatta (2025, 2026), Shin et al. (2025)
- Empirical/practitioner: Warren et al. (2025), Steensen et al. (2024), Cazzamatta (2025)
- Computational/technical: Kavtaradze (2024), Srba et al. (2025)
- Standards/specifications: W3C Credible Web CG (CCIV, Signals, Tech Report), Credibility Coalition (Zhang et al. 2018)
Sources are drawn from institutions across Europe, North America, and Asia. No single research group dominates the evidence base. The convergence across independent perspectives strengthens the overall finding.
Some sources appear in multiple queries (Vandenberghe 2025, Uscinski & Butler 2013, Warren et al. 2025) but are used for different evidence extracts relevant to each query's specific question.
Collection Gaps¶
| Gap | Impact | Mitigation |
|---|---|---|
| Non-English academic literature | May contain gap documentation or framework proposals not captured | Low impact — English is the dominant language for fact-checking methodology research |
| IFCN/EFCSN internal methodology documents | Potential unpublished evidence evaluation guidelines | Low-medium impact — public-facing codes are procedural, not methodological |
| Conference workshop papers | Framework proposals may exist in low-visibility venues | Low impact — significant proposals would be cited in the surveyed literature |
| Paywall-restricted full texts | Several papers accessible only through summaries | Low impact — consistent characterization across multiple access points |
Collection Self-Audit¶
| Domain | Rating | Notes |
|---|---|---|
| Eligibility criteria | Low risk | Clear criteria defined by each query's specific question |
| Search comprehensiveness | Low risk | 9 searches, 90 results dispositioned, 30 selected across 3 queries |
| Evaluation consistency | Low risk | Same scoring framework applied to all 16 sources |
| Synthesis fairness | Low risk | All hypotheses actively tested; H1 (positive) specifically sought |
Resources¶
Summary¶
| Metric | Value |
|---|---|
| Queries investigated | 3 |
| Files produced | 158 |
| Sources scored | 16 |
| Evidence extracts | 16 |
| Results dispositioned | 30 selected + 60 rejected = 90 total |
Tool Breakdown¶
| Tool | Uses | Purpose |
|---|---|---|
| WebSearch | 10 | Search queries across academic and standards literature |
| WebFetch | 14 | Page content retrieval (8 succeeded, 6 blocked by paywalls/403s) |
| Write | 80 | File creation for research archive |
| Read | 2 | Reading specification documents |
| Edit | 0 | No file modifications |
| Bash | 5 | Directory creation, batch file generation |
Token Distribution¶
| Category | Tokens |
|---|---|
| Input (context) | ~200,000 |
| Output (generation) | ~50,000 |
| Total | ~250,000 |