Abstract: Personalized medicine uses Clinical Role Models (CRMs)-individuals with similar health profiles-to build patient trust. However, CRM identification is challenged by data scarcity and privacy constraints. While Medical Large Language Models (MLLMs) can synthesize CRMs, they risk generating invalid "hallucinations" and face inconsistencies from evolving risk-scoring tools. We propose CASE (Constraint-Aware active SElfimproving fine-tuning), which integrates data-driven optimization with MLLM's generative capabilities to produce reliable CRMs. CASE employs a robust-optimization-based verifier to ensure CRMs are clinically valid, meeting patient requirements and reducing risk. Verified examples are then used to fine-tune the MLLM in a self-improving loop. We developed an active learning algorithm with utility-driven sampling and provide rigorous theoretical guarantees including high-probability sublinear regrets of the learning algorithm and patient safety guarantees of CASE. From clinical and survey data, we found that generated CRMs outperform real CRMs by >130% in reliability and 23% in Effort-to-Change reduction. We also found that a clear learning signal determines the outcomes of the fine-tuned model. Our work presents a novel integration of data-driven optimization and generative AI to enhance trust and decision-making in personalized medicine.