Statistical and Computational Challenges in Probabilistic Scientific Machine Learning (SciML)

We propose a data-driven method to learn the time-dependent probability densities of a multivariate stochastic process from sample paths, assuming that the initial probability density is analytically known. Our method uses a novel time-dependent binary classifier to approximate the partial time derivative of the logarithm of the probability density. We propose a contrastive learning-based objective to train the classifier. Significantly, the proposed method explicitly models the time-dependent probability distribution. We demonstrate that the proposed method can approximate the time-dependent probability density functions for systems driven by white noise. We also employ the proposed method for synthesizing new samples of a random vector from a given set of observations. In such applications, we generate sample paths for training using stochastic interpolants. Subsequently, new samples are generated using gradient-based Markov chain Monte Carlo methods because automatic differentiation can efficiently provide the necessary gradient. Further, we demonstrate the utility of an explicit approximation to the time-dependent probability density function through applications in unsupervised outlier detection. Our method accurately reconstructs complex time-dependent,  multi-modal, and near-degenerate densities, scales effectively to moderately high-dimensional problems, and reliably detects outliers among real-world data.

With Sobolev training, neural networks are provided data about both
the function of interest and its derivatives. This setting is
prevalent in scientific machine learning---appearing in molecular
dynamics emulators, derivative-informed neural operators, and
predictors of summary statistics of chaotic dynamical systems---as
well as in traditional machine learning tasks like teacher-student
model distillation. However, fundamental questions remain: How does
over-parameterization influence performance? What role does the
signal-to-noise ratio play? And is additional derivative data always
beneficial?

In this work, we study these questions using tools from statistical
physics and random matrix theory. In particular, we consider Sobolev
training in the proportional asymptotics regime in which the problem
dimensionality d, single hidden-layer features p, and training points
n grow to infinity at fixed ratios. We focus on target functions
modeled as single-index models (i.e., ridge functions with a single
intrinsic dimension), providing theoretical insights into the effects
of derivative information in high-dimensional learning.

Joint with Kate Fisher, Timo Schorlepp, and Youssef Marzouk.