Fine-tuning diffusion models is commonly carried out in practice using reinforcement learning algorithms such as Proximal Policy Optimization (PPO). Despite the remarkable empirical success of these approaches, the theoretical understanding of their convergence behavior remains rather limited. In this paper, we provide the first convergence guarantee for PPO-style algorithms for fine-tuning diffusion models. Specifically, we characterize the convergence rate of PPO in terms of two diffusion-specific factors that fundamentally govern RL-based fine-tuning: (i) the sampler stochasticity parameter $lambda$, which controls trajectory interpolation between deterministic and stochastic denoising dynamics, and (ii) the KL-regularization coefficient $mu$, which keeps the fine-tuned policy to remain close to the pretrained model. Our results imply that increased sampler stochasticity $lambda$, which corresponds to trajectories closer to DDPM-style sampling, is more favorable for RL fine-tuning, and stronger KL regularization (i.e., larger $mu$) provably accelerates convergence. Our experiments further validate our theoretical results.