(62g) Hybrid Gaussian Process Regression for Improved Predictability, Interpretability and Extrapolation

Gaussian Process Regression (GPR) is a powerful non-parametric tool used to approximate arbitrary continuous functions while also providing a measure of prediction uncertainty [1]. However, GPR treats the system as a black box, and consequently, it often results in poor extrapolation performance, lacks interpretability [2], and can violate underlying physics. Another contributing factor for the uncertainty in estimating posterior parameters can be attributed to the nonconvexity of the Maximum Likelihood Estimation (MLE) function [3] [4], which often leads to local optima and a strong dependence on the initial prior settings. To overcome these challenges, various techniques that embed physics-based knowledge in GPR are actively being investigated, including the development of special covariance functions that meet physical constraints [5], use of physics-based priors [6], incorporation of physics-based linear operator constraints into the covariance function [7], and construction of mean and covariance functions based only on the collection of realizations of the physics-based model [8]. In the area of physics-informed Neural Networks (PINNs), the training objective function is augmented by error terms that penalize physics violations, and this imposes the physics as “soft constraints” [9], while more rigorous optimization techniques have been proposed to identify optimal hyperparameters using duality theory [10]. However, NN surrogate models do not offer the significant advantage of GPR, namely the quantification of uncertainty, that is employed for adaptive exploration and design of experiments.

In this work, we propose a hybrid framework to obtain the best available posterior hyperparameters by utilizing all the accessible information or physics-based knowledge we have in hand, in addition to the given observations for training GPR. Here, the physics-informed posterior hyperparameters are estimated via penalizing the MLE function with additional error terms, which quantify the violation between physics-based knowledge and the GPR prediction. Our approach can incorporate various sources of physics-based information, including derivative- or integral-based, and known initial or boundary conditions. Within our hybrid modeling framework, we also take advantage of useful properties of Gaussian Processes (i.e., a linear transformation of a GP also follows a GP [1]).

Our results show that this approach can increase the prediction performance, while reducing the prediction uncertainty, when compared to standard GPR. We will show that this approach is a form of regularization, which leads to consistent convergence to more generalizable models, even when a warm-start initial prior estimate is not known. We perform a systematic analysis of the effect of embedding different physics information, on the map of the uncertainty provided by GPR models. Different mechanistic models in the form of partial differential equations (e.g., the heat equation) are approximated by hybrid GPRs and we present the prediction performance, uncertainty maps, consistency in parameters or interpretability, and computational cost. We also compare all the aforementioned results with black-box GPR models and other GPR hybridization techniques.

[1] Rasmussen, C.E. and C.K.I. Williams, Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). 2005: The MIT Press.

[2] Lin, L.-H. and V. Roshan Joseph, Transformation and Additivity in Gaussian Processes. Technometrics, 2020. 62(4): p. 525-535.

[3] Bayarri, M.J., et al., A Framework for Validation of Computer Models. Technometrics, 2007. 49(2): p. 138-154.

[4] Chen, Z. and B. Wang, How priors of initial hyperparameters affect Gaussian process regression models. Neurocomputing, 2018. 275: p. 1702-1710.

[5] Wahlström, N., et al. Modeling magnetic fields using Gaussian processes. in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. 2013.

[6] Särkkä, S. Linear Operators and Stochastic Partial Differential Equations in Gaussian Process Regression. 2011. Berlin, Heidelberg: Springer Berlin Heidelberg.

[7] Jidling, C., et al., Linearly constrained Gaussian processes. 2017.

[8] Yang, X., G. Tartakovsky, and A. Tartakovsky, Physics-Informed Kriging: A Physics-Informed Gaussian Process Regression Method for Data-Model Convergence. 2018.

[9] Raissi, M., P. Perdikaris, and G.E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 2019. 378: p. 686-707.

[10] Fioretto, F., et al., Lagrangian Duality for Constrained Deep Learning. 2021. p. 118-135.