(362p) Data-Driven Modeling of Complex Nonlinear Systems Using Hybrid Series and Parallel Nonlinear Static – Dynamic Stochastic Neural Networks

Developing accurate first-principles models for complex nonlinear stochastic dynamic systems can be time consuming, computationally expensive, and may be infeasible for certain systems due to lack of knowledge. It is also challenging to adapt first-principles models for time-varying probabilistic process systems. Data-driven or black-box models are relatively easier to develop, simulate and adapt online¹. However, it can be difficult to accurately represent complex, nonlinear stochastic dynamical system using data-driven models. The typical data-driven system identification approaches for nonlinear dynamic processes models comprise the Hammerstein and Wiener models, along with their many variants such as Hammerstein-Wiener and Wiener-Hammerstein type of models. Such models mostly consider a linear time invariant (LTI) dynamic model like transfer functions², Laguerre models³, etc. integrated with a nonlinear static model like polynomial functions⁴, neural networks⁵, etc. or the Gaussian radial basis function (RBF) kernel for the primal-dual formulation of least squares support vector machines^6,7 (LS-SVMs). These models may fail to accurately capture the nonlinearities in the process dynamics or the uncertainties in parameter estimates. Therefore, it is desired to consider both static and dynamic models to be nonlinear and probabilistic. However, the presence of nonlinearities in both the static and dynamic networks makes it considerably difficult to synthesize optimal hybrid network and estimate the network parameters. This work proposes the development of hybrid stochastic series and parallel static-dynamic neural networks with nonlinearities in both static and dynamic models. Efficient training algorithms are also developed for optimal synthesis of the network and estimation of parameters.

Conventional backpropagation algorithms for training static and dynamic neural networks use first order methods, but these methods may require significant tuning of hyper parameters, can suffer from slow convergence rates, and may not even converge for certain problems. On the contrary, second order methods can address some of these issues but can be subjected to excessive computational expense due to Hessian calculation, may be limited in terms of candidate architectures, and may only be used for estimating parameters during training of small to medium sized networks without incurring excessive computational expense. Therefore, applying the second-order methods for the overall nonlinear static-dynamic network in a monolithic approach can induce excessive computational expense. Moreover, classical Gaussian RBF with fixed centers may suffer from the curse of dimensionality for modeling higher order systems with larger input space and may be extremely sensitive to noisy data. Furthermore, the best optimization algorithm and its parameters for converging the static network model may be different than that for converging the dynamic network model. This work focuses on developing algorithms that enable training the hybrid probabilistic networks where the static and dynamic networks can be trained independently by different optimization algorithms, while solving an outer layer of optimization for estimating the connection weights between the static and dynamic models. Bayesian machine learning (ML) approaches are used for learning the probabilistic static model parameters. Both the series and parallel types of architecture have been considered to develop flexible models that offer tradeoff between computational expense and accuracy for highly nonlinear systems and are flexible for incorporating modifications in network architecture.

The proposed algorithms are applied to train the hybrid networks for three nonlinear dynamic processes – a pH neutralization reactor, the Van de Vusse reactor, and a pilot plant for post-combustion CO₂ capture using the monoethanolamine solvent⁸. It is observed that the hybrid series and parallel probabilistic static-dynamic models show superior performance compared to the existing state of the art LTI dynamic – nonlinear static models as well as the LS-SVM approaches, especially for the CO₂ capture system. In summary, the proposed algorithms show promise for solving large nonlinear dynamic stochastic network problems.

References

1. Su, H.-T., Bhat, N., Minderman, P. A. & McAvoy, T. J. Integrating Neural Networks With First Principles Models for Dynamic Modeling. Dynamics and Control of Chemical Reactors, Distillation Columns and Batch Processes (IFAC, 1993). doi:10.1016/b978-0-08-041711-0.50054-4.

2. Wigren, T. Recursive prediction error identification using the nonlinear wiener model. Automatica 29, 1011–1025 (1993).

3. Saha, P., Krishnan, S. H., Rao, V. S. R. & Patwardhan, S. C. Modeling and predictive control of MIMO nonlinear systems using Wiener-Laguerre models. Chem. Eng. Commun. 191, 1083–1119 (2004).

4. Biagiola, S. I. & Figueroa, J. L. Identification of uncertain MIMO Wiener and Hammerstein models. Comput. Chem. Eng. 35, 2867–2875 (2011).

5. Zhao, H., Guiver, J., Neelakantan, R. & Biegler, L. T. A nonlinear industrial model predictive controller using integrated PLS and neural net state-space model. Control Eng. Pract. 9, 125–133 (2001).

6. Goethals, I., Pelckmans, K., Suykens, J. A. K. & De Moor, B. Identification of MIMO Hammerstein models using least squares support vector machines. Automatica 41, 1263–1272 (2005).

7. Falck, T. et al. Least-Squares Support Vector Machines for the identification of Wiener-Hammerstein systems. Control Eng. Pract. 20, 1165–1174 (2012).

8. Chinen, A. S., Morgan, J. C., Omell, B., Bhattacharyya, D. & Miller, D. C. Dynamic Data Reconciliation and Validation of a Dynamic Model for Solvent-Based CO₂ Capture Using Pilot-Plant Data. Ind. Eng. Chem. Res. 58, 1978–1993 (2019).