(241c) Regularization and the Effect of Uncertainty in Parameter Estimation for NRTL Models

In chemical engineering, highly nonlinear models are abundant. Their quality hinges on proper identification of the underlying phenomena and reliable estimation of the model parameters. Examples are reaction kinetics, thermodynamic properties, e.g., activity coefficients obtained from NRTL, or mass transfer correlations.

Typically, least squares estimators are applied to obtain parameter estimates using real-life experimental data. The uncertainty of the parameter estimates is commonly calculated using successive linearizations, which result in the confidence ellipsoids frequently published alongside parameter estimates.

The linearization is also used for the analysis of model uncertainty and consequently for stochastic optimization. The assumptions underlying these linearizations do not in general hold for highly nonlinear models as, e.g., in chemical engineering (cf. Fig. 1a). Hence, parameter estimates of nonlinear models should be handled carefully, until a new way of estimation of parameter errors is established. At the same time, the combination of nonlinearity of these models, measurement uncertainty, and small number of measurements available frequently causes ill-conditioning of parameter estimation (PE) problems.

While engineers use rigorous or semi-rigorous equations to describe phenomena, they end up with models which are “overparametrized” for a system at hand: for example, they lack experiments at broader range of temperatures to describe the temperature dependence of a reaction rate. In such a case, estimation of parameters of an Arrhenius-type equation is impossible without regularization techniques, i.e., techniques ensuring conditioning and identifiability of the PE problems. In cases, where interdependence or correlation of parameters cannot be seen directly, regularization methods are essential.

Many regularization methods exist to solve these least squares problems. Among them are subset selection (Burth et al., 1999; Quaiser & Mönnigmann, 2009), Tikhonov regularization (Tikhonov & Arsenin, 1977), TSVD (Xu, 1998), and the orthogonalizations by Gram-Schmidt (Chu & Hahn, 2007) and Householder (Chen et al., 1989). All these regularizations imply a bias-error tradeoff through their hyperparameters.

In this contribution, we investigate the importance of proper experimental design and regularization and their effect on the solution of PE problems and uncertainty quantification of parameter estimates.

As a case study, the vapor liquid equilibrium (VLE) of 1-propanol and propyl-acetate is taken. The activity coefficients are described by the NRTL model, with five unknown parameters: two parameters for each component and the non-randomness parameter α.

In this example, the NRTL model cannot perfectly describe the VLE performance of the 1-propanol / propyl-acetate mixture, which is a frequent occurrence for complex nonlinear models in real-life engineering applications. Plant / model or data / model mismatch is inherent.

Measurement data both from a full factorial design – with three levels of granularity of 9, 15, and 27 experimental points – as well as from an optimal experimental design (OED), cf. (Vanaret et al., 2021), are applied to investigate the influence of the amount and quality of experimental data as well as the regularizations on the quality of the PE and the resulting prediction uncertainty.

Usually, analysis of the regularization techniques is done using artificially generated data. Here we base the entire analysis on actual experimental data, which is obtained in our own laboratory.

To analyze the effects on the performance of the least squares estimator, the amount of measurement data is varied, and different values are chosen for hyperparameters of the regularization methods mentioned above. At the same time, uncertainty estimates obtained by the Cramér-Rao bound (Fréchet, 1943) are compared against the ones calculated by Monte Carlo simulations.

In Fig. 1, preliminary results hereof are shown. On the right-hand side (b), the influence of selecting different subsets of identifiable parameters can be observed: For the 1-propanol / propyl-acetate mixture three parameters of the NRTL model were estimated: one binary interaction parameter for each component and α. Using regularization by subset selection (Quaiser, 2009), parameter is removed from the PE problem. This runs contrary to what is usually done in thermodynamics, where is typically fixed instead. Fig.1 (b) depicts both cases. Here, fixing leads to a larger confidence ellipsoid, i.e., larger uncertainty, compared to the solution suggested by the regularization.

Regarding PE in general, the results show that regularization is required to solve the ill-conditioned least squares formulations. Unidentifiable parameters lead to more solver iterations and badly conditioned Hessians near the solution. At the same time, the regularization leads to more accurate results and more reliable estimates regarding parameter uncertainty quantification. The identifiable subset of parameters has in general a lower quantifiable uncertainty than the larger set.

Regarding OED, the regularizations in PE problems may help in reducing the parameter space for the OED problems, which can lower computational and experimental cost.

In future work, we will more closely examine how these regularizations affect the iterative cycle of PE, OED, and real-life experiments.

References

Bates, D.M., Watts, D.G.: Nonlinear Regression Analysis and Its Applications, Wiley, April 2007, ISBN 978-0-470-13900-4.

Burth, M. Verghese, G.C., Vélez-Reyes, M.: Subset selection for improved parameter estimation in on-line identification of a synchronous generator. IEEE Transactions on Power Systems, 14(1):218–225, 1999.

Xu., P.: Truncated SVD methods for discrete linear ill-posed problems. Geophysical Journal International, 135:505–514, 1998.

Quaiser, T., Mönnigmann, M.: Systematic identifiability testing for unambiguous mechanistic modeling – application to JAK-STAT, MAP kinase, and NF-kB signaling pathway models. BMC Syst Biol. 3, 50, 2009.

Tikhonov, A.N., Arsenin, V.Y.: Solutions of ill-posed problems. V. H. Winston, Washington, USA, 1977.

Chu, Y., Hahn, J.: Parameter set selection for estimation of nonlinear dynamic systems. AIChE Journal, 53(11):2858–2870, 2007.

Chen, S., Billings, S.A., Luo, W.: Orthogonal least squares methods and their application to nonlinear system identification, International Journal of Control, 50:5, 1873-1896, 1989.

Vanaret, C., Seufert, P., Schwientek, J., Karpov, G., Ryzhakov, G., Oseledets, I., Asprion, N., Bortz, M.: Two-phase approaches to optimal model-based design of experiments: how many experiments and which ones?, Computers & Chemical Engineering, 46, 107218, 2021.

Fréchet, M.: Sur l'extension de certaines evaluations statistiques au cas de petits echantillons, Revue de l'Institut International de Statistique, 11(3/4), 182-205, 1943.

Figure 1: (a) comparison of a 95% confidence ellipses estimated by linearization and actual uncertainty of parameters calculated using Monte Carlo simulations (left); (b) comparison of 95% confidence ellipses that depend on a set of selected parameters (right).