Reinforcement learning has become a core ingredient of LLM post-training, but much of today’s pipeline is built around an unusually narrow learning signal: the model is optimized to maximize scalar reward, often derived from little more than whether an output succeeds. In this talk, I will present two recent directions for going beyond this paradigm. 

The first part introduces Maximum Likelihood Reinforcement Learning (MaxRL), which starts from the observation that in correctness-based domains the model implicitly defines a likelihood over successful rollouts, while standard RL optimizes only a lower-order approximation to that likelihood. Empirically, MaxRL consistently outperforms standard RL baselines across the models and tasks studied, delivers up to 20× gains in test-time scaling efficiency, and shows stronger scaling with additional data and compute. 

The second part introduces Reinforcement Learning from Text Feedback (RLTF), a learning paradigm in which natural-language critiques are available during training but not at inference, requiring the model to internalize richer feedback rather than merely condition on it. Together, these works suggest that the next frontier of LLM reinforcement learning lies not only in scaling optimization, but in rethinking both the objectives we optimize and the forms of supervision we learn from.