Q003 — Self-Audit¶


Research	R0021 — Prompt engineering definitions
Run	2026-03-25
Query	Q003

ROBIS 4-Domain Audit¶

Domain 1: Eligibility Criteria¶

Rating: Low risk

Criterion	Assessment
Criteria defined before searching	Yes — "quantifiable" defined as including numbers/thresholds/testable criteria
Criteria stable during research	Yes
Criteria appropriate to query	Yes

Notes: The definition of "quantifiable" was established before analysis began.

Domain 2: Search Comprehensiveness¶

Rating: Some concerns

Criterion	Assessment
Multiple search strategies used	Yes — separate searches per vendor
Searches designed to test each hypothesis	Partially — searched for recommendations, not for absence
All results dispositioned	Yes
Source diversity achieved	Yes — all four named vendors covered

Notes: Limited to primary documentation pages. Vendor cookbooks, blog posts, and tutorials may contain additional quantifiable guidance not captured here.

Domain 3: Evaluation Consistency¶

Rating: Low risk

Criterion	Assessment
All sources scored using same framework	Yes
Evidence typed consistently	Yes — all Factual
ACH matrix applied	Yes
Diagnosticity analysis performed	Yes

Notes: Same quantifiable/subjective classification applied consistently across vendors.

Domain 4: Synthesis Fairness¶

Rating: Some concerns

Criterion	Assessment
All hypotheses given fair hearing	Yes — Google's quantifiable elements acknowledged
Contradictory evidence surfaced	Yes — Google as partial counterexample
Confidence calibrated to evidence	Yes
Gaps acknowledged	Yes

Notes: The classification of "quantifiable" vs. "subjective" involves judgment. Structural recommendations (XML tags, prompt chaining) are actionable but were classified as non-quantifiable. This classification could be debated.

Overall Assessment¶

Overall risk of bias: Some concerns

The main risk is in the classification judgment. The researcher's hypothesis that "prompt engineering is not engineering" benefits from finding subjective guidance. The classification was applied consistently but could be critiqued.

Researcher Bias Check¶

Confirmation bias: Strong risk. The finding aligns with the researcher's thesis. Mitigated by using a clear, pre-defined classification criterion and acknowledging borderline cases (structural recommendations).
Framing bias: "Quantifiable" vs. "subjective" is itself a frame that emphasizes what's missing rather than what's present. Alternative framings (e.g., "actionable" vs. "vague") might produce different results.