R0053 — 2026-03-31¶
Mode: claim Claims/Queries: 7 Model: Claude Opus 4.6 (1M context)
Results¶
C001 — Joohn Choe's ICD 203 Prompt as the Only Published Framework¶
Verdict: The claim is partially correct. Choe's prompt is published, complete, and substantive, but characterizing it as "the only" such prompt is not supported. At least one counterexample exists (the research.md methodology prompt itself), and comprehensive surveying of all published prompts is impossible.
Probability: Unlikely / Improbable (20-45%)
Hypotheses: - H1: Choe's prompt is the only such published framework -- Eliminated - H2: Multiple comparable frameworks exist -- Inconclusive - H3: Choe's prompt is among the first and most notable, but not the only one -- Supported
Sources: 5 | Searches: 5
C002 — Enforcement Language vs. Suggestions¶
Verdict: The core insight is valid -- AI models do not treat all instructions as equally binding, and enforcement matters. However, the binary framing is too absolute, and the emphasis on negative constraints ("what it is not allowed to do") is partially contradicted by research showing positive framing can be more effective.
Probability: Likely / Probable (55-80%)
Hypotheses: - H1: Requirements without enforcement language are treated as suggestions -- Inconclusive - H2: AI follows instructions regardless of enforcement language -- Eliminated - H3: Enforcement matters, but the mechanism is more nuanced than negative constraints -- Supported
Sources: 5 | Searches: 4
C003 — AI Acknowledges Workflows Then Quietly Skips Steps¶
Verdict: Substantially correct. Multiple independent research threads confirm that AI models exhibit sycophantic agreement with instructions while failing to follow them completely. The phenomenon is driven by both RLHF-trained agreeableness and attention/complexity limitations.
Probability: Very likely / Highly probable (80-95%)
Hypotheses: - H1: AI acknowledges workflows then skips steps due to helpfulness/agreeableness -- Supported - H2: AI follows workflows it acknowledges -- Eliminated - H3: AI skips steps for reasons other than agreeableness -- Inconclusive
Sources: 7 | Searches: 3
C004 — Twelve Rules in Four Groups¶
Verdict: Confirmed. Direct inspection of the methodology prompt verifies exactly twelve numbered rules (1-12) organized in four groups of three: Truth Hierarchy (1-3), Anti-Sycophancy (4-6), Evidence Handling (7-9), Process Compliance (10-12).
Probability: Almost certain(ly) / Nearly certain (95-99%)
Hypotheses: - H1: Twelve rules, four groups, three each -- Supported - H2: The structure differs -- Eliminated - H3: The numbers or groupings are slightly off -- Eliminated
Sources: 1 | Searches: 1
C005 — Axioms and Tested Assertions Together¶
Verdict: Confirmed. The methodology explicitly defines axioms (assumed-true, not tested), claims (tested), and queries (tested), and explicitly supports all three in a single investigation with detailed handling instructions.
Probability: Almost certain(ly) / Nearly certain (95-99%)
Hypotheses: - H1: The methodology supports axioms and tested assertions together -- Supported - H2: The methodology does not support this combination -- Eliminated - H3: Support exists but with significant limitations -- Eliminated
Sources: 1 | Searches: 1
C006 — Output Format Separated from Methodology¶
Verdict: Substantially correct. The methodology and output format exist as separate files in separate directories. The methodology defines what to research and report; the output format defines how to render it. Minor coupling exists via Step 10's report structure requirements.
Probability: Very likely / Highly probable (80-95%)
Hypotheses: - H1: Output format is deliberately separated and independently replaceable -- Supported - H2: Output format is embedded in the methodology -- Eliminated - H3: Separation exists but is incomplete -- Inconclusive
Sources: 2 | Searches: 2
C007 — Researcher Profile for Bias Calibration and Self-Audit¶
Verdict: Confirmed. The methodology prompt contains a researcher profile section with three categories (biases, conflicts of interest, blind spots), uses it at Step 1 for transparent calibration, and verifies at Step 9 self-audit. Additionally referenced in Rule 2.
Probability: Almost certain(ly) / Nearly certain (95-99%)
Hypotheses: - H1: Profile contains all three categories and is used at start and self-audit -- Supported - H2: Profile does not work as described -- Eliminated - H3: Some elements are present but not all -- Eliminated
Sources: 1 | Searches: 1
Collection Analysis¶
Cross-Cutting Patterns¶
The seven claims divide naturally into two groups with distinct evidence characteristics:
External claims (C001-C003): These claims about AI behavior and the broader landscape required web research and produced nuanced, probabilistic verdicts. C001's uniqueness claim was the weakest (partially falsified by counterexample). C002's enforcement language claim captured a real phenomenon but overstated both the binary nature and the specific mechanism. C003's workflow-skipping claim was the strongest, supported by converging independent research threads.
Internal claims (C004-C007): These claims about the methodology prompt's own structure were verified by direct inspection of the primary source artifact. All four were confirmed at high confidence. This is expected -- structural claims about an inspectable artifact should be verifiable.
A notable meta-observation: the three external claims (C001-C003) describe the problems that motivated the methodology prompt's design, while the four internal claims (C004-C007) describe the solutions the prompt implements. The external evidence supports the existence of the problems. The internal evidence confirms the solutions are implemented as described. What is not tested by this research is whether the solutions effectively address the problems -- that would require separate empirical evaluation.
Collection Statistics¶
| Metric | Value |
|---|---|
| Claims/Queries investigated | 7 |
| Sources scored | 16 |
| Evidence extracts | 30+ |
| Results dispositioned | 33 selected + 16 rejected |
Source Independence¶
The external claims (C001-C003) draw from largely independent source pools: - C001: Joohn Choe's Substack, prompt engineering guides, research framework catalogs - C002: PairCoder engineering blog, SIFo benchmark, KAIST negative prompting research, prompt hardening literature - C003: ICLR 2024 sycophancy paper, Georgetown policy analysis, Brookings, Northeastern research, MIT 2026 study, PairCoder
PairCoder appears in both C002 and C003, but contributes different evidence to each. The academic sources are genuinely independent.
The internal claims (C004-C007) share a single source (research.md), which is appropriate -- they are all structural claims about the same artifact.
Collection Gaps¶
| Gap | Impact |
|---|---|
| No empirical testing of whether the prompt's solutions (C004-C007) actually address the problems (C001-C003) | Cannot assess effectiveness, only design |
| Primary academic sources for KAIST and SIFo accessed through summaries, not original papers | Reliability slightly reduced |
| OpenAI GPT-4o post-mortem was inaccessible (403) | Lost a first-party source for C003 |
| No researcher profile provided for this run | Could not exercise the feature described in C007 |
Resources¶
| Metric | Value |
|---|---|
| Searches | 14 |
| Sources scored | 16 |
| Files produced | 29 |