New Directions in Reinforcement Learning and Control

Learning in dynamical systems is a fundamental challenge underlying modern sequence modeling. Despite extensive study, efficient algorithms with formal guarantees for general nonlinear systems have remained elusive. This talk presents a provably efficient framework for learning in bounded and Lipschitz nonlinear dynamical system, establishing the first sublinear regret guarantees in a dimension-free setting. Our approach combines Koopman lifting, Luenberger observers, and spectral filtering to show that nonlinear dynamics are learnable. These insights motivate a new neural architecture, the Spectral Transform Unit (STU), which achieves state-of-the-art performance on language modeling, dynamical system, and differential equation benchmarks.

Reward functions and reinforcement learning have staged a return. Dismissed as relics of an age before data hyper-scaling, they prove increasingly potent in a world that grows more compute-rich than data-rich. This talk presents a trilogy of results on the renewed power of reward-based methods.

In the first part, we address a puzzle from the realm of large-scale preference fine-tuning: why does Reinforcement Learning from Human Feedback typically outperform the seemingly more natural Direct Policy Optimization? The supervised path, it turns out, leads somewhere darker than expected.

In the second, we consider Inverse Reinforcement Learning as a remedy for the compounding errors that afflict Behavioral Cloning — mistakes that, like a fellowship astray, multiply with each step taken off the true distribution. IRL, through the structural power of a reward class, achieves guarantees that interactive methods require an expert oracle to match, and does so without one.

In the third, we turn to test-time compute scaling. Leveraging the submodular structure of deployment metrics such as pass@k and best@k, we propose a novel RL method for optimizing generative models that generalizes and illuminates the recent MaxRL algorithm. Remarkably, beyond enabling scalable inference, it often achieves superior results even on single-sample output — suggesting that one objective function, properly wielded, may yet rule them all

The evolution of AI from passive prediction to agentic decision-making, rooted in RL and control and now accelerated by LLMs, surfaces new generalization challenges: deployed agents must cope with never-before-seen conditions, and execute tasks that themselves may not have been seen, all while obeying safety requirements. In this talk I will use linear-quadratic control as a theoretical testbed for studying two important aspects of generalization in AI agents. First, I will address generalization across initial states, showing that the extent to which it succeeds under policy gradient training depends on the degree of exploration induced by the system when commencing from initial states seen in training. Second, I will turn to generalization across tasks, establishing that it is more difficult to achieve with safety requirements (formalized as $H_infty$-robustness) than without, regardless of how well safety requirements are met on tasks seen in training. Experiments demonstrate that the conclusions of the analyses extend to non-linear control with a neural network agent, and to customer relationship management with an LLM agent. Our findings suggest that common efforts to enhance generalization in AI agents may be insufficient, and that learning new representations is a promising alternative.

The adjacent fields of reinforcement learning (RL) and adaptive control share the same objectives, yet they are separated by a wide cultural gap. In this presentation, I attempt to bridge this gap for the linear quadratic regulator (LQR) problem, which serves as a cornerstone and the benchmark for both fields. I begin by discussing different learning pipelines, including direct and indirect (model-based) approaches, as well as episodic and online (adaptive) approaches. Despite the extensive literature spanning several decades, numerous problems remain unsolved. For instance, RL methods are seldom concerned with closed-loop stability certificates or efficient implementations, while the adaptive control community has dedicated minimal effort to optimality. We address the data-driven LQR problem in an adaptive setting, which entails online recursive algorithms and closed-loop data, and we seek both algorithmic as well as closed-loop certificates. Our approach encompasses different variations of policy gradient methods and employs a novel covariance parameterization of the LQR problem. Finally, all our theoretical results are validated through simulations and experiments in diverse domains, demonstrating the computational and sample efficiency of our method.

Data-driven and learning-based methods have attracted considerable attention in recent years both for the analysis of dynamical systems and for control design. While there are many interesting and exciting results in this direction, our understanding of fundamental limitations of learning for control is lagging. This talk will focus on the question of when learning can be hard or impossible in the context of dynamical systems and control. In the first part of the talk, I will discuss a new observation on immersions and how it reveals some potential limitations in learning Koopman embeddings. In the second part of the talk, I will show what makes it hard to learn to stabilize linear systems from a sample-complexity perspective. While these results might seem negative, I will conclude the talk with some thoughts on how they can inspire interesting future directions.

A major problem in the study of large language models is to understand their inherent low-dimensional structure. We introduce an approach to study the low-dimensional structure of language models at a model-agnostic level: as sequential probabilistic models. We first empirically demonstrate that a wide range of modern language models exhibit low-rank structure: in particular, matrices built from the model’s logits for varying sets of prompts and responses have low approximate rank. Taking a theoretical perspective, we then show that any distribution over sequences with such structure of low approximate logit rank can be provably learned using polynomially many queries to the model's logits and polynomial time. Finally, we show that insights resulting from this perspective of low-rank can be leveraged for generation— for instance, we can generate a response to a target prompt using a linear combination of the model’s outputs on unrelated, or even nonsensical prompts. Further, we show how such insights can explain phenomena observed in fine-tuning of LLMs, namely those relating to subliminal learning.

The process of discovery requires active exploration—the act of collecting new and informative data. However, efficient autonomous exploration remains a major unsolved problem. The dominant paradigm addresses this challenge by using Reinforcement Learning (RL) to train agents with intrinsic motivation, maximizing a composite objective of extrinsic and intrinsic rewards. We suggest that this approach incurs unnecessary overhead: while policy optimization is necessary for precise task execution, employing such machinery solely to expand state coverage may be inefficient.

In this talk, we propose a new paradigm that explicitly separates exploration from exploitation and bypasses RL during the exploration phase. Our method uses a tree-search strategy inspired by the Go-With-The-Winner algorithm, paired with a measure of epistemic uncertainty to systematically drive exploration. By removing the overhead of policy optimization, our approach explores an order of magnitude more efficiently than standard intrinsic motivation baselines on hard Atari benchmarks. Further, we will demonstrate that this unsupervised exploration mechanism generates high cumulative reward trajectories that can successfully be distilled into robust executable policies using existing supervised backward learning algorithms.