(362f) Development of Bayesian Machine Learning Algorithms for Optimal Nonlinear Model Selection and Parameter Estimation from Noisy Data of Unknown Characteristics

Process models are very essential in the study of dynamics, prediction of outputs and control of process plants. While data driven models serve as a powerful alternative to high fidelity first-principles models which are often complicated and require considerable resources to develop, the selection of an appropriate model form for highly nonlinear systems is a significant challenge. The model should be sparse and provide succinct representation of a system and should possess favorable characteristics for being used in optimization. A classical model structure that has proven very useful for many systems is the state space model. However, these models can yield large errors for highly nonlinear systems. One simple extension of the state space model for nonlinear systems is to include bilinear and extended bilinear terms. In our earlier work, such models were presented and an algorithm was developed for parameter estimation under uncertainty for systems with noisy data with unknown noise characteristics. A two-stage Bayesian machine learning (ML) algorithm was developed that was found to be very effective not just in the prediction of model behaviors but also for exploring the connectivity structures of process plants [1]. However, the model form may be restrictive for highly nonlinear systems. In addition, sparsity was not guaranteed.

Recently, several authors have employed sparse regression for succinct representation of dynamical systems through optimal selection of basis function from several candidate basis functions[2][3]. However, the algorithms are applicable to the data without noise and therefore the efficacy of the developed algorithms were tested using simulated data. Furthermore, the candidate basis functions were rather limited. However, most real-life process data are corrupted with noise with unknown characteristics. Besides, there may be colored noise present in the data.

In this work, we develop a Branch and Bound algorithm [4] for subset selection using an information criteria that rewards model fitness while penalizing model size and model complexity. Large number of basis functions are considered including polynomial, logarithmic, and exponential functions, and interaction terms between inputs and outputs including time delays this leading to a large-scale combinatorial optimization problem. A desired number of hierarchically ranked candidate models are obtained leaving the user with the choice of selecting any top rank model that yields succinct representation of the system with explainable basis functions. Parameters are estimated by employing Bayesian inferencing by maximizing the likelihood of the estimated model parameters. The two-step approach also estimates the unknown noise characteristics in an outer loop conditioned on available data. The method facilitates incorporating prior user belief of model parameters. Based on an expectation maximization algorithm, the algorithm yields simultaneous estimation of the noise covariance in addition to the model parameters.

The proposed algorithm is applied to the simulated data for a reactor-separator system and also to the operational data from an industrial boiler. The algorithm shows promise for flexible and succinct model development for highly nonlinear systems from data with correlated noise and unknown noise characteristics.

References

[1] T. Bankole and D. Bhattacharyya, “Exploiting connectivity structures for decomposing process plants,” J. Process Control, vol. 71, pp. 116–129, 2018, doi: 10.1016/j.jprocont.2018.09.002.

[2] B. Bhadriraju, M. S. F. Bangi, A. Narasingam, and J. S. Il Kwon, “Operable adaptive sparse identification of systems: Application to chemical processes,” AIChE J., vol. 66, no. 11, pp. 1–15, 2020, doi: 10.1002/aic.16980.

[3] N. M. Mangan, J. N. Kutz, S. L. Brunton, and J. L. Proctor, “Model selection for dynamical systems via sparse regression and information criteria,” Proc. R. Soc. A Math. Phys. Eng. Sci., vol. 473, no. 2204, 2017, doi: 10.1098/rspa.2017.0009.

[4] A. A. H. Land and A. G. Doig, “Automatic Method of Solving Discrete Programming Problems”. JSTOR vol. 28, no. 3, pp. 497–520, 1960.