Abstract: Deep learning models, whether serving as surrogate representations of high-fidelity simulations or directly informed by physical data, have become indispensable for alleviating the computational bottlenecks inherent in the inference and optimization tasks associated with digital twins of complex physical systems. This reliance emphasizes the critical need for rigorous validation and uncertainty quantification methods to guide model construction and facilitate trustworthy deployment in high-consequence decision-making processes. In this work, we present the Occam Plausibility Algorithm for Surrogate models (OPAL-surrogate), a systematic framework for discovering “optimal” deep learning surrogate models across the expansive model space, encompassing diverse deep learning classes, architectures, and hyperparameters. The framework leverages hierarchical Bayesian inference for the principled determination of both network parameters, hyperparameters, and model plausibility, complemented by model validation under uncertainty to assess prediction accuracy and reliability. Adhering to these principles OPAL-surrogate provides an efficient strategy for adaptively balancing the trade-off among model complexity, accuracy, and prediction uncertainty. We demonstrate the effectiveness of OPAL-surrogate through two illustrative modeling problems: the deformation of porous materials for building insulation and the simulation of turbulent combustion flow governing shear-induced ablation of solid fuels in hybrid rocket motors. Further extensions to the framework in digital twin settings, including its application to the selection of “right” models for patient-specific brain tumor treatment and for controlling the nano-manufacturing processes of self-assembled materials, will also be discussed.