This was part of Reinforcement Learning from Offline Data and Human Feedback

Stochastic Zeroth-Order Policy Optimization for RLHF

Lei Ying, University of Michigan