R0049/2026-03-31/Q001-SRC04 — Scorecard¶
Source¶
| Title | LLM SATs FTW |
| Publisher | sroberts.io (personal blog) |
| Authors | Scott Roberts |
| Date | 2025 |
| URL | https://sroberts.io/posts/llm-sats-ftw/ |
| Type | Blog post with published source code |
Ratings¶
| Dimension | Rating |
|---|---|
| Reliability | Medium |
| Relevance | High |
| Missing data | Low risk |
| Measurement | Some concerns |
| Selective reporting | Some concerns |
| Randomization | N/A |
| Protocol deviation | N/A |
| COI/funding | Low risk |
Rationale¶
| Dimension | Rationale |
|---|---|
| Reliability | Blog post with source code, not peer-reviewed; practitioner perspective with hands-on implementation |
| Relevance | Directly implements intelligence community structured analytic techniques as LLM-powered tools — the most relevant finding for Q001 |
| Missing data | Source code available; methodology described in detail |
| Measurement | No quantitative evaluation of technique effectiveness; qualitative assessment only |
| Selective reporting | Author presents positive results; limited discussion of failures or limitations beyond "LLMs are not a replacement for human analysts" |
| COI/funding | Independent practitioner, no apparent conflicts |
Evidence Extracts¶
| Evidence | Summary |
|---|---|
| SRC04-E01 | Roberts implemented ACH, Starbursting, and Key Assumptions Check as separate LLM-powered Streamlit applications with published source code |