(126e) Recurrent Neural Networks, Numerical Integrators and Nonlinear System Identification

Based on older work [1-4] we formulate the problem of identifying nonlinear ODEs and PDEs from time-dependent data as a recurrent neural net templated on established numerical simulation algorithms: initial value problem solvers, like Runge-Kutta methods for ODEs, and method of lines for PDEs.

"Unrolled" feedforward as well as recurrent architectures (with shared parameters) are used – more importantly, we consider implementations templated on implicit integrators (like the Crank-Nicolson), and explore the ability of these architectures to learn relations (and not just functions).

We couple these methods with nonlinear manifold learning techniques (giving us data-driven variables in terms of which the ODEs/PDEs are formulated) and illustrate the approach by identifying nonlinear effective PDEs arising in the study of large ensembles of heterogeneous networks. In particular, we study models of the neuronal network of the SCN (the suprachiasmatic nucleus, responsible for controlling circadian rhythms).

Finally, we note connections of the form of our learned network--with an explicitly included differential equation initial value solver--to popular recurrent network architectures of less specific form, such as the gated recurrent unit (GRU) or long short-term memory (LSTM) networks.

[1] Rico-Martínez, R., & Kevrekidis, I. G. (1995). Nonlinear system identification using neural networks: dynamics and instabilities. In A. B. Bulsari (Ed.), Neural Networks for Chemical Engineers (pp. 409–442). Elsevier.
[2] González-García, R., Rico-Martínez, R., & Kevrekidis, I. G. (1998). Identification of distributed parameter systems: A neural net based approach. Computers & Chemical Engineering, 22(98), S965–S968.
[3] Rico-Martínez, R., Kevrekidis, I. G., Kube, M. C., & Hudson, J. L. (1993). Discrete- vs continuous-time nonlinear signal processing attractors, transitions and parallel implementation issues. In American Control Conference (pp. 1475–1479).
[4] Anderson, J. S., Kevrekidis, I. G., & Rico-Martínez, R. (1996). A comparison of recurrent training algorithms for time series analysis and system identification. Computers & Chemical Engineering, 20(96), S751–S756.