(363f) Machine Learning and System Identification for Dynamical Systems: A Comparative Review

Machine Learning has been long hailed as a panacea for various challenges in many disciplines. While the promises of machine learning have indeed advanced the state of the art in various fields (ranging from computer vision [1] to advertising and marketing [2]), there has been comparatively limited research on the application of machine learning to the modelling of dynamical systems [3]. The data-driven modelling of dynamical systems, classically referred to as system identification, represents an important class of problems for the engineering community. This is because, as a community, we are often interested in developing and devising optimal operating strategies for our systems of interest. This requires knowledge of the dynamic and transient behaviour of the system. Consequently, this knowledge, in the form of a model, can be used to develop model-based control and optimization algorithms, with Model Predictive Control (MPC) being an example of such an approach.

Additionally, the limited research that has been performed in this area has left much to be desired with a mixture of results being reported [4,5]. On one hand, some hail machine learning as the answer to a number of the recurring challenges within system identification, while others cast a more doubtful view. This mixed opinion is similarly echoed in the related field of time series analysis and forecasting, where a variety of researchers have demonstrated traditional approaches outperforming more complex machine learning methods [6]. However, while this may be the case, there are clear gaps within the methodologies used to compare different classes of models. For example, it is often not evident how model selection is performed. Indeed, this represents the crux of the challenge of selecting an appropriate model for the problem at hand. Without a rigorous model selection procedure, it is difficult to make any conclusions about model performance. In addition to this, the comparison of model performance between different model classes is often limited to measures of “model accuracy” alone. While this may be appropriate for the types of problems addressed within different disciplines, such one-dimensional measures of performance are limiting when considering control and optimization applications. For example, a given model may be more accurate than another but if training and/or inference time prohibit real-time applications then its performance for control and optimization applications may be dubious at best.

For the above reasons, we propose a thorough and rigorous comparative review of the application of traditional system identification approaches and modern machine learning methods to the modelling of dynamical systems. We select several promising methods from both sets of approaches and test their performance on a variety of dynamical systems prevalent within the process systems engineering community and beyond. To address the aforementioned limitations, we propose a pipeline for model selection, training, validation, and subsequent comparison. Additionally, we utilize a variety of model performance criteria beyond the standard metric of “model accuracy” alone. This includes measures of “computational complexity” such as training and inference time as well as measures relevant to control and optimization such as online model-adaptation time. We also consider the performance of the different methods in the various small and big data limits while testing the robustness of the approaches to varying degrees of measurement noise. The full pipeline deployed in the comparative study is depicted in Fig. 1.

Our comparative study highlights that a binary approach to classifying a given method, tool, or technology as either “good” or “bad” adds little value. Instead, a nuanced approach is required, which our study provides. For example, we found that some methods, such as Gaussian Processes, were more adept at handling limited amounts of data in comparison to others. Consequently, such methods may be useful to compensate for situations where limited amounts of high-quality data are available, which is often the case in industry. At the same time, we found that other methods, such as Neural Ordinary Differential Equations, were able to achieve high model accuracy but required long training and inference times thus nullifying their use for real-time applications. To summarise, we provide a taxonomy of many system identification and machine learning methods with respect to various modelling criteria for the modelling of dynamical systems. Our work can therefore be used as a benchmark for new system identification algorithms and as a means to compare a variety of different approaches under a single holistic framework.

References

[1] Voulodimos, A., Doulamis, N., Doulamis, A. and Protopapadakis, E., 2018. Deep Learning for Computer Vision: A Brief Review. Computational Intelligence and Neuroscience, 2018, pp.1-13.

[2] Kietzmann, J., Paschen, J. and Treen, E., 2018. Artificial Intelligence in Advertising. Journal of Advertising Research, 58(3), pp.263-267.

[3] Ljung, L., 2008. Perspectives on System Identification. IFAC Proceedings Volumes, 41(2), pp.7172-7184.

[4] Chiuso, A. and Pillonetto, G., 2019. System Identification: A Machine Learning Perspective. Annual Review of Control, Robotics, and Autonomous Systems, 2(1), pp.281-304.

[5] Ljung, L., Andersson, C., Tiels, K. and Schön, T., 2020. Deep Learning and System Identification. IFAC-PapersOnLine, 53(2), pp.1175-1181.

[6] Makridakis, S., Spiliotis, E. and Assimakopoulos, V., 2018. Statistical and Machine Learning forecasting methods: Concerns and ways forward. PLOS ONE, 13(3), p.e0194889.