(59ad) Reinforcement Learning (RL)-Based Process Controller Design: An Implementable Approach

Reinforcement learning (RL)-based techniques show promising potentials in a variety of process control applications [1, 2]. In standard model-free RL-based approach, a significant amount of data, coming from online interactions between an agent (controller) and an environment (process), is required to train the controller. This, in turn, may not only be costly but also put process safety at risk, rendering it unimplementable in practice. To address this issue, several RL-based methods have been proposed using a pre-trained RL agent which is trained offline using a surrogate model [3]. Even though these works have proposed excellent ideas for RL-based control designs, the implementation of these techniques remain impractical in many real-world situations because of the need for developing sufficiently accurate models.

On the other hand, a limited amount of information (in the form of step-test data or simple linear model), is only available in many practical situations to develop a control strategy. Therefore, the need for an accurate model in model-based optimal control approaches, such as model predictive control (MPC), and the promising performance of RL-based controllers raise the question of how to safely develop a model-free RL controller using this minimal information available in practice? In this direction, some researchers have addressed the problem of PID tuning using RL strategy [4]. A step-response model is employed to train an agent offline. The agent is then fine-tuned through online interactions with the actual process. Although this study suggests a great way to put the RL algorithm to use in practice, its implementation can only be limited to single-input single-output (SISO) systems.

Motivated by the above considerations, this work addresses the problem of safe implementation of RL controller by leveraging information from already implemented MPCs (developed using simple linear models). To that end, the existing MPC formulation is employed to compute control actions offline (based on a wide range of process conditions). This information is then used to pre-train an RL agent with a performance that replicates the MPC controller. In the next step, the pre-trained agent is used within a model-free RL framework to interact with the actual process so as to improve its performance over the MPC controller. Finally, the ability of the proposed RL controller to handle uncertainty in the process parameters is demonstrated. The effectiveness of the proposed approach is illustrated using a CSTR example.

References

[1] Spielberg, S., Tulsyan, A., Lawrence, N.P., Loewen, P.D. and Bhushan Gopaluni, R. (2019). Toward self‐driving processes: A deep reinforcement learning approach to control. AIChE journal, 65(10), e16689.

[2] Dogru, O., Wieczorek, N., Velswamy, K., Ibrahim, F. and Huang, B. (2021). Online reinforcement learning for a continuous space system with experimental validation. Journal of Process Control, 104, 86-100.

[3] Brandi, S., Piscitelli, M.S., Martellacci, M. and Capozzoli, A. (2020). Deep reinforcement learning to optimise indoor temperature control and heating energy consumption in buildings. Energy and Buildings, 224, 110225.

[4] Dogru, O., Velswamy, K., Ibrahim, F., Wu, Y., Sundaramoorthy, A.S., Huang, B., Xu, S., Nixon, M. and Bell, N. (2022). Reinforcement learning approach to autonomous PID tuning. Computers & Chemical Engineering, 161, 107760.