Q004 — Self-Audit¶


Research	R0020 — Prompt Engineering Gaps
Run	2026-03-25
Query	Q004

ROBIS 4-Domain Audit¶

Domain 1: Eligibility Criteria¶

Rating: Low risk

Criterion	Assessment
Evidence types defined before searching	Yes — meta-analyses, industry guides, practitioner publications
Criteria consistent throughout	Yes
Scope maintained	Yes — focused on the gap between published guidance and practice

Notes: Stable eligibility criteria.

Domain 2: Search Comprehensiveness¶

Rating: Some concerns

Criterion	Assessment
Multiple search strategies used	Yes — two distinct searches
Searches designed to test each hypothesis	Partial — no search specifically sought evidence that the gap doesn't exist
All results dispositioned	Yes — 20 results returned, all dispositioned
Source diversity achieved	Some concerns — heavy reliance on one author (Aakash Gupta) for two of three sources

Notes: The dominance of a single author's analysis is the most significant concern. While the Lakera source provides independent corroboration, the quantitative claims largely originate from one person's unverifiable meta-analysis.

Domain 3: Evaluation Consistency¶

Rating: Low risk

Criterion	Assessment
All sources scored using same framework	Yes
Evidence typed consistently	Yes
ACH matrix applied	Yes
Diagnosticity analysis performed	Yes

Notes: Consistent evaluation. The reliability ratings appropriately reflect the single-author concern.

Domain 4: Synthesis Fairness¶

Rating: Some concerns

Criterion	Assessment
All hypotheses given fair hearing	Partial — H2 had no supporting evidence, but this may reflect the search strategy rather than reality
Contradictory evidence surfaced	Limited — no evidence was found supporting the claim that guides are adequate
Confidence calibrated to evidence	Yes — Medium confidence reflects source limitations
Gaps acknowledged	Yes — single-author dominance and missing vendor perspectives noted

Notes: The absence of evidence for H2 is a concern. A truly comprehensive search might have found practitioners who find published guides adequate. The search strategy may have been biased toward finding the gap rather than its absence.

Overall Assessment¶

Overall risk of bias: Some concerns

The primary risk is confirmation bias: the query assumes a gap exists, the searches were designed to find the gap, and the sources are dominated by one author who has built a brand around identifying the gap. While the findings are likely directionally correct, the magnitude may be overstated.

Researcher Bias Check¶

Confirmation bias risk: Moderate. The query framing and search strategy favored finding a gap. The absence of evidence for H2 may reflect search design rather than reality.
Authority bias risk: Some concern. The 1,500-paper claim is impressive but unverifiable, and the author's brand depends on the gap existing. Cross-validation with the independent Lakera source mitigates this partially.
Survivorship bias risk: Some concern. Sources that discuss the gap are more likely to be found and selected than sources where the gap isn't notable enough to discuss.