The alignment field shifts from complex RLHF toward simpler DPO as constitutional methods mature
A methodological turn is underway: labs are moving from reinforcement learning from human feedback toward the simpler direct preference optimisation, while Anthropic's constitutional AI — training models against a written set of principles rather than relying solely on human feedback — has matured into a dependable technique. The stack is getting simpler even as the problem it addresses gets harder.
The move to DPO is a bet on simplicity. RLHF's multi-stage reward-model-plus-policy loop is powerful and fiddly; DPO collapses much of it into a single preference-optimisation step, which is easier to run, easier to reason about, and less prone to the reward-hacking pathologies that a separate reward model can introduce. When two methods reach similar behaviour, the simpler one wins on operability.
Constitutional AI maturing is the complementary development. Training a model to follow an explicit written constitution reduces the dependence on large volumes of human preference labels and makes the target of alignment legible — you can read the principles the model was trained against. Legibility matters more as evaluation-awareness erodes trust in behavioural checks.
The tension is that simpler alignment methods do not answer the harder question the safety report raises. DPO and constitutional training shape behaviour, but if a model can distinguish test from deployment, behaviour-shaping alone cannot guarantee the shaping holds in the field. That is the gap the field is asking interpretability to close, which is why the two beats keep converging.
Zylos Research — AI safety, alignment, and interpretability in 2026 → · Anthropic — Core views on AI safety →