(740c) Optimal Regularization Strategies for Large-Scale Parameter Estimation Problems
AIChE Annual Meeting
2019
2019 AIChE Annual Meeting
Computing and Systems Technology Division
Process Modeling and Control Applications I
Thursday, November 14, 2019 - 4:08pm to 4:27pm
Ill-posed estimation problems arise in applications due to the presence of a large number of parameters, lack of informative experimental data, and large measurement errors. These issues give rise to parameter estimates that are unstable (they exhibit extreme sensitivity to measurement noise). In other words, the estimated parameters exhibit high variance when exposed to measurement noise. To deal with such problems, one often uses regularization strategies either in the form of prior cost terms or constraints [1]. Cost (prior) regularization is the most common regularization approach but tuning the associated weight matrix is non-trivial [2]. Specifically, one most carefully balance fitting error and variance.
In this work, we present an alternative approach for regularizing ill-posed estimation problems that is optimal for linear models and that can be easily adapted to deal with nonlinear problems. In seminal work, Park [3] proved that for linear models with near rank deficiencies, one can design parameter constraints by using eigenvector s of the input covariance matrix that minimize parameter variance. In other words, no other set of constraints can decrease the variance further. Moreover, this approach provides a mechanism to embed prior information, by identifying parameter combinations that can be estimated from the data. This approach has been used in kinetic modeling [4, 5].
We propose a computational approach that regularizes general estimation problems by performing an eigenvalue decomposition of the parameter reduced Hessian matrix. We first note that the reduced Hessian matrix can be computed in a scalable manner by using random perturbations of the measurements [6,7]. We then compute eigenvectors of the reduced Hessian matrix associated to small eigenvalues are used this to formulate regularizing constraints. We demonstrate that this approach, while not optimal for nonlinear problems, performs well in practice. Moreover, we show that this approach can be used to determine optimal weights for cost regularization terms and to guide the design of trust region constraints.
References
[1] A Gabor and J R Banga. Robust and efficient parameter estimation in dynamic models of biological systems. BMC systems biology, 9(1):74, 2015.
[2] F Bauer and M A Lukas. Comparingparameter choice methods for regularization of ill-posed problems. Mathematics and Computers in Simulation, 81(9):1795–1841, 2011.
[3] S H Park. Collinearity and optimal restrictions on regression parameters for estimating responses. Technometrics, 23(3):289–295, 1981.
[4] S Vajda, H Rabitz, E Walter, and Y Lecourtier. Qualitative and quantitative identifiability analysis of nonlinear chemical kinetic models. Chemical Engineering Communications, 83(1):191–219, 1989.
[5] S Vajda, P Valko, and T Turanyi. Principal component analysis of kinetic models. International Journal of Chemical Kinetics, 17(1):55–81, 1985.
[6] Y Bard. Nonlinear Parameter Estimation. Academic Press, 1974.
[7] S Shin, O Venturelli, and V M Zavala. Scalable nonlinear programming framework for parameter estimation in dynamic biological system models. PLoS computational biology, In Press, 2019.