Uncertainty Quantification and Machine Learning for Complex Physical Systems

Computer simulations are essential for exploring input-output relationships in engineering and science but can be computationally expensive for extensive what-if analyses. Gaussian process emulators offer a powerful statistical approach to approximating simulations, but their scalability is often hindered by the costly inversion of large correlation matrices. To overcome this challenge, we introduce new methods leveraging the random Fourier feature technique from computer science to accelerate Gaussian process emulators. Our approach enhances computational efficiency while maintaining accuracy, making it suitable for a broad range of simulations, including those with gradient information, functional outputs, and stochastic outputs. Through numerical experiments, we demonstrate that our methods outperform existing ones in speed and accuracy, with theoretical results validating these improvements.

Measurements and observations in particle physics fundamentally depend on one's ability to quantify their uncertainty and, thereby, their significance. Therefore, as machine learning (ML) methods become more prevalent in high energy physics, being able to determine the uncertainties of an ML method becomes more important. A wide range of possible approaches has been proposed, however, there has not been a comprehensive comparison of individual methods. To address this, the Fair Universe project organized the HiggsML Uncertainty Challenge, which took place from September 2024 to March 2025, and the dataset and performance metrics of the challenge will serve as a permanent benchmark for further developments. Additionally, the Challenge was accepted as an official NeurIPS2024 competition. The goal of the challenge was to measure the Higgs signal strength, using a dataset of simulated $pp$ collision events observed in LHC. Participants were evaluated on both their ability to precisely determine the correct signal strength, as well as on their ability to report correct and well-calibrated uncertainty intervals. In this talk, we present an overview of the competition itself and of the infrastructure that underpins it. Further, we present the winners of the competition and discuss their winning uncertainty quantification approaches.

 

The HiggsML Uncertainty Challenge itself can be found under https://www.codabench.org/competitions/2977/

And more details are available as https://arxiv.org/abs/2410.02867

Consider an expensive computer simulation of a complex system whose inputs are governed by a known distribution and whose output demarcates passing and failing.  For example, we are motivated by a simulation of airflow around an aircraft wing with inputs specifying wing design/flight conditions and output indicating whether aerodynamic efficiency standards have been met.  Our ultimate objective is to quantify the probability of system failure (which could be rare) with only several hundred evaluations of the expensive simulation.  We tackle this problem in three parts.  First, we develop a Bayesian deep Gaussian process (DGP) surrogate to furnish predictions and uncertainty quantification at unobserved inputs.  DGPs outperform ordinary GPs when dynamics are nonstationary.  Second, we propose a contour locating sequential design scheme to train the DGP to identify the failure contour in the response surface.  Third, we incorporate a hybrid Monte Carlo estimator of the failure probability which combines DGP surrogate predictions with strategically allocated evaluations of the expensive model.  All methods are supported by publicly available software and the “deepgp” R package on CRAN.

Astrostatistics is a growing interdisciplinary field at the interface of astronomy and statistics. Astronomy is a field rich with publicly available data, but inference using these data must acknowledge selection effects, measurement uncertainty, censoring, and missingness. In the Astrostatistics Research Team (ART) at the University of Toronto --- a joint team between the David A. Dunlap Department of Astronomy & Astrophysics and the Department of Statistical Sciences --- we take an interdisciplinary approach to analysing astronomical data from a range of objects such as stars, star clusters, and galaxies. In this talk, I will cover three ART projects that employ Bayesian inference techniques to: (1) find stellar flares in time series data from stars using hidden Markov models, (2) investigate the uncertain relationship between old star cluster populations and their host galaxies using hurdle models, and (3) discover potential "dark" galaxies within an inhomogeneous Poisson Process framework using noisy data.

Software testing is essential for the reliable and robust development of complex software systems. A key step in software testing is fault localization, which uses test data to pinpoint failure-inducing combinations for further diagnosis. Existing fault localization methods have two key limitations: they (i) do not incorporate domain and/or structural knowledge from test engineers, and (ii) do not provide a probabilistic assessment of risk for potential root causes. Such methods can thus fail to confidently whittle down the combinatorial number of potential root causes in complex systems, resulting in prohibitively high testing costs. To address this, we outline a holistic machine-learning-guided test case design and fault localization (MaLT) framework, which leverages recent probabilistic machine learning methods to accelerate the testing of complex software systems. MaLT consists of three steps: (i) the construction of a suite of test cases using a covering array for initial testing, (ii) the investigation of posterior root cause probabilities via a Bayesian fault localization procedure, then (iii) the use of such Bayesian analysis to guide selection of subsequent test cases via active learning. The proposed MaLT framework can thus facilitate efficient identification and subsequent diagnosis of software faults with limited test runs.

In many applications, including climate-model emulation and calibration, there is a need to learn the conditional distribution of a high-dimensional spatial field given a covariate vector, based on a small number of training samples. We propose a nonparametric Bayesian method that decomposes this challenging conditional density estimation task into a large series of univariate autoregressions that we model using heteroskedastic Gaussian processes with carefully chosen prior parameterizations. We describe scalable variational inference based on stochastic gradient descent. The resulting generative model can be used to sample from the learned distribution or transform existing fields as a function of the covariate vector. We provide numerical illustrations and comparisons on simulated data and climate-model output.

This presentation focuses on a novel methodological framework, called the “Actor-Simulator,” that incorporates the calibration of digital twins into model-based reinforcement learning for more effective control of stochastic systems with complex nonlinear dynamics. Traditional model-based control often relies on restrictive structural assumptions (such as linear state transitions) and fails to account for parameter uncertainty in the model. These issues become particularly critical in industries such as biopharmaceutical manufacturing, where process dynamics are complex and not fully known, and only a limited amount of data is available. Our approach jointly calibrates the digital twin and searches for an optimal control policy, thus accounting for and reducing model error. We balance exploration and exploitation by using policy performance as a guide for data collection. This dual-component approach provably converges to the optimal policy, and outperforms existing methods in extensive numerical experiments based on the biopharmaceutical manufacturing domain.

Gaussian processes are commonly used tools for modeling continuous processes in machine learning and statistics. This is partly due to the fact that one may employ a Gaussian process as an interpolator for a finite set of known points, which can then be used for prediction and straight forward uncertainty quantification at other locations. However, it is not always the case that the available information is in the form of a finite collection of points. For example, boundary value problems contain information on the boundary of a domain, which is an uncountable collection of points that cannot be incorporated into typical Gaussian process techniques. We propose and construct Gaussian processes that unify, via reproducing kernel Hilbert space, the typical finite case with the case of having uncountable information by exploiting the equivalence of conditional expectation and orthogonal projections. We show existence of the proposed Gaussian process and that it is the limit of a conventional Gaussian process conditioned on an increasing but finite number of points. We illustrate the applicability via numerical examples.

In this talk, I will introduce a novel framework for physics-informed debiasing of machine learning estimators, which we call Simulation-Calibrated Scientific Machine Learning (SCaSML). This approach leverages the structure of physical models to achieve three key objectives: - Unbiased Predictions: It produces unbiased predictions even when the underlying machine learning predictor is biased. - Overcoming Dimensionality Challenges: It mitigates the curse of dimensionality that often affects high-dimensional estimators. - Inference Time Scaling: Improve the machine learning estimation by allocating inference time computation. The SCaSML paradigm integrates a (potentially) biased machine learning algorithm with a de-biasing procedure that is rigorously designed using numerical analysis and stochastic simulation. We dynamically refines and debiases the SCiML predictions during inference by enforcing the physical laws. Our methodology aligns with recent advances in inference-time computation—similar to those seen in the large language model literature—demonstrating that additional computation can enhance ML estimates. Furthermore, we establish a surprising equivalence between our framework and another research direction that utilizes approximate (linearized) solvers to precondition iterative methods. This connection not only bridges two distinct areas of study but also offers new insights and algorithms into improving estimation accuracy in physics-informed machine learning settings.

Inverse scattering aims to infer information about a hidden object by using the received scattered waves and training data collected from forward mathematical models. Recent advances in computing have led to increasing attention towards functional inverse inference, which can reveal more detailed properties of a hidden object. However, rigorous studies on functional inverse, including the reconstruction of the functional input and quantification of uncertainty, remain scarce. Motivated by an inverse scattering problem where the objective is to infer the functional input representing the refractive index of a bounded scatterer, a new Bayesian framework will be discussed in the first part of this talk. It contains a surrogate model that takes into account the functional inputs directly through kernel functions, and a Bayesian procedure that infers functional inputs through the posterior distribution. In the second part of this talk, we will introduce a novel procedure that, given sparse data generated from a stationary deterministic nonlinear dynamical system, can characterize specific local and/or global dynamic behavior with rigorous probability guarantees. More precisely, the sparse data is used to construct a statistical surrogate model based on a Gaussian process (GP). The dynamics of the surrogate model is interrogated using combinatorial methods and characterized using algebraic topological invariants (Conley index). The GP predictive distribution provides a lower bound on the confidence that these topological invariants, and hence the characterized dynamics, apply to the unknown dynamical system.