Research	R0040 — RLHF Alternatives
Run	2026-03-29
Query	Q001

Q001 — RLHF Alternatives — Query Definition

Query as Received

What alternatives to RLHF are being considered or in use by the AI research community?

Query as Clarified

• Subject: Methods for aligning large language models that differ from or improve upon Reinforcement Learning from Human Feedback (RLHF)
• Scope: Both academic research proposals and production-deployed alternatives; covers changes to the optimization algorithm (DPO, GRPO), feedback source (RLAIF, RLVR), and training methodology (Constitutional AI, self-play)
• Evidence basis: Peer-reviewed papers, pre-prints from major AI labs, and industry deployment evidence

Ambiguities Identified

1. "Alternatives": Could mean complete replacements or incremental improvements. We interpret broadly to include both.
2. "Being considered": Could mean purely theoretical or actively deployed. We cover both, distinguishing between them.
3. "AI research community": Could mean academic only or include industry labs. We include both.
4. "RLHF" boundary: Some methods (e.g., RLAIF) retain the RL framework but change the feedback source. Whether these qualify as "alternatives" depends on how narrowly one defines RLHF.

Sub-Questions

1. What methods change the optimization algorithm while keeping human preference data? (DPO, KTO, IPO, ORPO, SimPO)
2. What methods change the feedback source away from human annotation? (RLAIF, Constitutional AI, RLVR)
3. What methods change the RL optimizer itself? (GRPO, GSPO)
4. What methods eliminate RL entirely? (DPO, KTO, self-play)
5. Which alternatives are in production use vs. research-only?

Hypotheses

Hypothesis	Statement	Status
H1	Multiple viable alternatives to RLHF exist and are in active use	Supported
H2	RLHF remains dominant with no viable alternatives	Eliminated
H3	RLHF is being augmented and specialized rather than replaced	Partially supported