Reinforcement Learning from Offline Data and Human Feedback

In dynamic decision-making scenarios across business and healthcare, leveraging sample trajectories from diverse populations can significantly enhance reinforcement learning (RL) performance for specific target populations. While existing transfer learning methods primarily focus on linear regression settings, they lack direct applicability to reinforcement learning algorithms. This paper pioneers the study of transfer learning for dynamic decision scenarios modeled by nonstationary finite-horizon Markov decision processes, utilizing neural networks as powerful function approximators and backward inductive learning. We demonstrate that naive sample pooling strategies, effective in regression settings, fail in Markov decision processes. To address this challenge, we introduce a novel “re-weighted targeting procedure” to construct “transferable RL samples” and propose “transfer deep Q-learning”, enabling neural network approximation with theoretical guarantees. We assume that the reward functions are transferable and deal with both situations in which the transition densities are transferable or nontransferable. Our analytical techniques for transfer learning in neural network approximation and transition density transfers have broader implications, extending to supervised transfer learning with neural networks and domain shift scenarios. Empirical experiments on both synthetic and real datasets corroborate the advantages of our method. (Joint work with Jinhang Chai and Elynn Chen)

We develop a stochastic control framework for fine-tuning diffusion models, using denoising diffusion probabilistic models as the pre-trained reference dynamics and combining linear dynamics control with Kullback–Leibler regularization. We establish well-posedness and regularity of the resulting control problem and propose a policy iteration algorithm, PI-FT, for its numerical solution. We prove that PI-FT converges globally at a linear rate and, unlike existing analyses that impose regularity assumptions throughout training, show that the iterates themselves preserve the required regularity. If time allows, we will also discuss how incorporating a risk-sensitive control criterion can better align the fine-tuned model with rare but important tail events in generation.

Group relative policy optimization (GRPO), a core methodological component of DeepSeekMath and DeepSeek-R1, has emerged as a cornerstone for scaling reasoning capabilities of large language models. Despite its widespread adoption and the proliferation of follow-up works, the theoretical properties of GRPO remain less studied. This paper provides a unified framework to understand GRPO through the lens of classical U-statistics. We demonstrate that the GRPO policy gradient is inherently a U-statistic, allowing us to characterize its mean squared error (MSE), derive the finite-sample error bound and asymptotic distribution of the suboptimalitygap for its learned policy. Our findings reveal that GRPO is asymptotically equivalent to an oracle policy gradient algorithm – one with access to a value function that quantifies the goodness of its learning policy at each training iteration – and achieves asymptotically optimal performance within a broad class of policy gradient algorithms. Furthermore, we establish a universal scaling law that offers principled guidance for selecting the optimal group size. Empirical experiments further validate our theoretical findings, demonstrating that the optimal group size is universal, and verify the oracle property of GRPO.

Pairwise human feedback is now widely used in both LLM alignment and LLM evaluation, from reward modeling in RLHF to public leaderboards based on head-to-head comparisons. However, these data are noisy, heterogeneous, and highly non-uniform, making uncertainty quantification a central statistical challenge. In this talk, I will present two recent works on this topic. The first studies reward learning under heterogeneous human feedback, jointly modeling latent rewards and annotator rationality, with asymptotic guarantees that enable valid reward comparison and uncertainty-aware best-of-N sampling. The second studies LLM evaluation as inference on a low-rank latent score tensor observed through pairwise comparisons, leading to efficient debiased inference and a score-whitening method for handling anisotropic information under non-uniform sampling. Together, these works illustrate how statistical inference can provide principled uncertainty quantification for both alignment and evaluation of large language models.

I will present a new algorithmic framework and analysis for off-policy evaluation (OPE) in finite-horizon MDPs. The algorithm learns a scalar weight for each data point by a moment matching objective against a discriminator class F that realizes Qπ. Notably, the theoretical guarantee of the algorithm is dimension-free, that the finite-sample error does NOT depend on the statistical complexity of the function class F (e.g., no log|F| dependence), and generalizes the standard error bound for linear regression with a fixed design. The algorithm is also closely connected to several existing methods, such as linear FQE, (sequential) importance sampling, and trajectory stitching, providing connections and novel perspectives to the foundational task of OPE.

Can offline reinforcement learning with function approximation ever be as easy as supervised learning? In general, the answer is no — the Bellman operator contracts only in the sup-norm, not in the L^2-norm induced by the data distribution. This geometric mismatch makes model-free value learning with function approximation provably harder than regression. However, real-world problems often come with additional structures that may facilitate reinforcement learning.In this talk, I will discuss recent advances in understanding the structures that enable model-free offline RL. Focusing on controlled Markov diffusions—a widely used class of dynamical systems—I will provide an affirmative answer to the question above. Specifically, I will identify ellipticity as a key structure that makes model-free RL with function approximation tractable with offline data. Leveraging ellipticity, I will demonstrate desirable geometric properties of Bellman operators in an appropriate Sobolev space. Based on these insights, I will introduce a new class of algorithms for model-free RL with function approximation that achieve near-optimal oracle inequalities efficiently. Finally, I will discuss an application to fine-tuning diffusion-based generative models, where the ellipticity structure is exploited to design a PDE-based algorithm that attains fast convergence rates.

We investigate the problem of offline policy learning in the presence of unobserved confounders, which may arise in both observational studies and adaptive experiments (e.g., self-selection and noncompliance in sequential medical settings). In particular, we study this problem under the f -sensitivity model, which characterizes the confounding effect by its “average” strength. Under the f -sensitivity model, we characterize the distribution shift from the observable to the counterfactual and design a distributionally robust policy learning algorithm, f -SR(ad)L, which maximizes the expected outcome within a given policy class Π . We show that the sub-optimality gap of f -SR(ad)L learned from a sequential (i.i.d. or adaptively collected) dataset is of the order O(κ(Π)n​), where κ(Π) is the entropy integral of Π under the Hamming distance and n is the sample size. A matching lower bound of is provided to show the optimality of the rate. Finally, we assess our method on synthetic and a real-world data on lung cancer treatments to demonstrate its advantage over existing benchmarks.

Fine-tuning diffusion models is commonly carried out in practice using reinforcement learning algorithms such as Proximal Policy Optimization (PPO). Despite the remarkable empirical success of these approaches, the theoretical understanding of their convergence behavior remains rather limited. In this paper, we provide the first convergence guarantee for PPO-style algorithms for fine-tuning diffusion models. Specifically, we characterize the convergence rate of PPO in terms of two diffusion-specific factors that fundamentally govern RL-based fine-tuning: (i) the sampler stochasticity parameter $lambda$, which controls trajectory interpolation between deterministic and stochastic denoising dynamics, and (ii) the KL-regularization coefficient $mu$, which keeps the fine-tuned policy to remain close to the pretrained model. Our results imply that increased sampler stochasticity $lambda$, which corresponds to trajectories closer to DDPM-style sampling, is more favorable for RL fine-tuning, and stronger KL regularization (i.e., larger $mu$) provably accelerates convergence. Our experiments further validate our theoretical results.