(333g) Deep Reinforcement Learning Approach for Process Control

In the recent years, there has been significant progress in the fields of computer vision and

natural language processing that followed the success of deep learning [1]. Human level control

has been attained in games [2] and physical tasks [3] by combining deep learning with

reinforcement learning resulting in 'Deep Q Network' [2]. In the process industry, Model

Predictive Control (MPC) has been found to be an effective control strategy. However,

application of MPC on nonlinear stochastic systems can be computationally demanding and may

require estimation of hidden states in a complex system. The performance of MPC depends

significantly on the quality of model and hence any Model Plant Mismatch (MPM) would be

detrimental to the performance. In this work, we use deep learning and reinforcement learning

for controlling process variables. We present an actor-critic based approach to deterministic policy gradient

reinforcement learning algorithm for control. During the training phase of the learning algorithm,

we consider the standard reinforcement learning setup, where an agent (controller) interacts with

an environment (process) through control actions and receives a reward in discrete time steps.

The training is done with full state observation. Deep neural networks serve as function

approximators and are used to learn the control policies. Once trained, the learned network

acquires a policy that maps system output to control actions and it can be used to control the

plant without full state observations. Since online optimization is not required as in MPC our

algorithm is computationally less intensive during online operation phase. We evaluated our

approach on Single Input Single Output Systems and Multiple Input Multiple Output Systems

with varying set points and initial conditions and compared it with MPC.

References:

[1] Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep

convolutional neural networks. In Advances in neural information processing systems, pp.

1097â??1105, 2012.

[2] Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel,

Bellemare, Marc G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg,

et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529â??533,

2015.

[3] Timothy P. Lillicrap , Jonathan J. Hunt , Alexander Pritzel, Nicolas Heess, Tom Erez,

Yuvalassa, David Silver & Daan Wierstra, Continuous control with deep reinforcement learning,

International Conference on Learning Representations, 2016.