Reinforcement learning has driven strong gains in LLM reasoning on static tasks. However, when applied to agents, they consistently fail in unfamiliar environments, where effective exploration is required. In this talk, we identify a failure pattern: as agents move out of distribution, their reasoning trajectories become progressively shorter, and exploration collapses. We show that standard entropy-based metrics are insufficient, and introduce mutual information as a more reliable signal of input-dependent behavior. Through a SNR perspective, I explain why low reward variance causes input-agnostic regularization to dominate, driving this collapse. We then ask what enables robust exploration. Through VAGEN, we demonstrate that learning a structured world model, decomposed into state estimation and transition dynamics, provides the necessary inductive bias, enabling a 3B VLM to outperform GPT-5 on agent benchmarks. Finally, we show that when this structure is learned matters. With Self-Play, we find that acquiring these priors via self-play before RL is significantly more effective than learning them during RL.
Overall, this suggests a simple paradigm: learn how the world works first, then learn what to do, with mutual information as a key diagnostic to prevent reasoning collapse.