Q001 — Self-Audit¶


Research	R0050 — Journalism and Other Truth-Seeking Disciplines
Run	2026-03-31
Query	Q001

ROBIS 4-Domain Audit¶

Domain 1: Eligibility Criteria¶

Rating: Low risk

Criterion	Assessment
Evidence criteria defined before searching	Yes — four specific structured elements defined by the query
Criteria applied consistently	Yes — each source evaluated against all four elements
Scope appropriate	Yes — all named organizations plus additional discovered sources

Notes: The eligibility criteria were well-defined by the query itself. The four elements (evidence hierarchy, calibrated uncertainty, bias assessment, source tiering) provided clear evaluation targets.

Domain 2: Search Comprehensiveness¶

Rating: Some concerns

Criterion	Assessment
Multiple search strategies used	Yes — 4 search operations across different organizations
Searches designed to test each hypothesis	Yes — searched for both presence and absence of elements
All results dispositioned	Yes — 21 results selected or rejected with rationale
Source diversity achieved	Partial — all sources are published methodology documents; no practitioner interviews or ethnographic studies

Notes: The search covered all named organizations and discovered additional sources (Verification Handbook). However, the search was limited to published methodology documents. Internal practices and practitioner knowledge were not accessible through web search. Wire services (Reuters, AP, AFP) may have more structured internal practices.

Domain 3: Evaluation Consistency¶

Rating: Low risk

Criterion	Assessment
All sources scored using same framework	Yes — same reliability/relevance/bias dimensions
Evidence typed consistently	Yes — Factual and Analytical types used consistently
ACH matrix applied	Yes — all evidence evaluated against all hypotheses
Diagnosticity analysis performed	Yes — most and least diagnostic evidence identified

Notes: Consistent evaluation across all six sources.

Domain 4: Synthesis Fairness¶

Rating: Low risk

Criterion	Assessment
All hypotheses given fair hearing	Yes — H1 received partial support through NewsGuard and PolitiFact
Contradictory evidence surfaced	Yes — NewsGuard as genuine formal system was acknowledged despite H3 being supported
Confidence calibrated to evidence	Yes — High confidence justified by strong convergence across independent sources
Gaps acknowledged	Yes — practitioner knowledge and internal practices flagged as gaps

Notes: The assessment explicitly acknowledges the researcher bias concern about IC framing and notes that journalism may have its own forms of structure.

Domain 5: Source-Back Verification¶

Rating: Low risk

Source	Claim in Assessment	Source Actually Says	Match?
SRC01	PolitiFact has six-level Truth-O-Meter but no evidence hierarchy	Methodology page confirms six ratings and implicit (not formal) source preferences	Yes
SRC02	NewsGuard has 9-criterion 0-100 scoring	Methodology page confirms 9 criteria, weighted scoring, pass-fail, and tiered ratings	Yes
SRC03	IFCN lacks all four structured elements	Five commitments contain no hierarchical scales, calibrated language, bias domains, or tiering	Yes
SRC04	BBC uses informal source categories	Guidelines use advisory language about source types without formal scoring	Yes
SRC05	Verification Handbook emphasizes principles over frameworks	Chapter explicitly states principles over prescriptive rules	Yes
SRC06	Bellingcat emphasizes case-by-case methodology	Published resources emphasize methodology as skill with case-by-case assessment	Yes

Discrepancies found: 0

Corrections applied: None needed

Unresolved flags: None

Notes: All source characterizations in the assessment accurately reflect the source content. The distinction between "outlet rating" (NewsGuard) and "evidence quality hierarchy" was correctly maintained.

Overall Assessment¶

Overall risk of bias: Low risk

The research process was well-structured with clear criteria, comprehensive search across all named organizations, and consistent evaluation. The main limitation is the reliance on published methodology documents rather than practitioner knowledge.

Researcher Bias Check¶

IC framing bias (declared): Directly relevant. The query frames journalism through IC criteria, which inherently highlights what journalism lacks rather than what it has. The assessment compensates by noting that journalism's principle-based approach is a deliberate choice, not a deficiency.
Completeness bias (declared): Mildly relevant. The finding that journalism lacks most of these elements could satisfy the researcher's desire to show the nine-framework methodology is comprehensive. However, the assessment honestly identifies that journalism may have its own forms of structure not captured by the query's framing.
Confirmation risk: Low. The evidence strongly and consistently supports H3, and the assessment acknowledges the partial support for H1.