Reward functions and reinforcement learning have staged a return. Dismissed as relics of an age before data hyper-scaling, they prove increasingly potent in a world that grows more compute-rich than data-rich. This talk presents a trilogy of results on the renewed power of reward-based methods.

In the first part, we address a puzzle from the realm of large-scale preference fine-tuning: why does Reinforcement Learning from Human Feedback typically outperform the seemingly more natural Direct Policy Optimization? The supervised path, it turns out, leads somewhere darker than expected.

In the second, we consider Inverse Reinforcement Learning as a remedy for the compounding errors that afflict Behavioral Cloning — mistakes that, like a fellowship astray, multiply with each step taken off the true distribution. IRL, through the structural power of a reward class, achieves guarantees that interactive methods require an expert oracle to match, and does so without one.

In the third, we turn to test-time compute scaling. Leveraging the submodular structure of deployment metrics such as pass@k and best@k, we propose a novel RL method for optimizing generative models that generalizes and illuminates the recent MaxRL algorithm. Remarkably, beyond enabling scalable inference, it often achieves superior results even on single-sample output — suggesting that one objective function, properly wielded, may yet rule them all