(194b) Distributed Reinforcement Learning for Nonlinear Control of Large-Scale Processes with Guaranteed Stability

The proliferation of large-scale chemical industrial systems in modern society highlights the critical need for effective control methodologies to ensure operational safety and efficiency [1][2]. These systems, such as refineries, fertilizer plants, and water treatment plants, form the backbone of numerous industries, contributing to essential products and services. However, the scale and complexity of these infrastructures present challenges in terms of control, particularly as they become increasingly interconnected and dynamic. In response to the growing complexity and interconnectedness of large-scale chemical industrial systems, there is an instant demand for distributed control strategies[3][4]. Unlike traditional centralized approaches, distributed control distributes decision-making across multiple subsystems, enabling more robust and adaptable operation in the face of system-wide disturbances or failures, which is particularly suitable for large-scale systems where individual components or processes may operate separately but still contribute to the performance of the overall system. In this way, distributed strategies can mitigate single points of failure and enhance the system's resilience to faults [5][6].

In the realm of networked process industries, Distributed Model Predictive Control (DMPC) is a prevalent method for managing complex systems with multiple interacting subsystems[7]. For instance, Stewart et al. [8] demonstrate that cooperative distributed control can achieve stability, constraint adherence, and Pareto optimality equivalent to centralized control in scenarios where subsystem coupling is sparse, leveraging sub-optimal control theory. With technological advancements, distributed or decentralized MPC approaches have been applied in diverse fields including electric vehicle charging scheduling [9], energy generation and storage schemes [10], and car-platoon control [11]. Nevertheless, distributed MPC exhibits several drawbacks. Firstly, it necessitates precise first-principles models, which can pose a significant challenge. Additionally, DMPC struggles to accommodate dynamic system changes effectively. Moreover, issuing control signals at high frequencies in real-time poses difficulties. Lastly, for nonlinear systems, achieving sub-optimal solutions often requires numerous iterations, imposing substantial computational burdens, particularly in the context of large-scale systems.

An alternative method for complex system controlling is Reinforcement learning (RL), where the controller updates its strategy through the interaction with the environment and targets maximizing its rewards by trial and error, which learns from interaction with the environment and adapts its control policies over time [12][13]. However, little works address the stability issue for distributed RL, where each subsystem relies solely on local information for decision-making. Moreover, the concurrent updating of policies across multiple subsystems introduces stationary concerns, as the environment of one subsystem, which contains other subsystems, keeps changing over time. In response to these challenges, new control methods tailored specifically for distributed RL are imperative to ensure the security of the global system.

In this paper, we present a novel distributed RL framework with a stability guarantee to ensure the stable operation of large-scale chemical industrial systems. Within this framework, we establish sub-actor-critic controllers for each subsystem and introduce a communication protocol which exchanges a bit of information during the training process and limits communication during execution. The stability of the global system is guaranteed with the utilization of Lyapunov conditions. Such a distributed communication protocol alleviates the real-time operational burden of distributed systems. Further, we validate the efficacy of our algorithm in ensuring stable control performance across two distinct chemical simulation systems.

References

• Daoutidis P., Zachar M., Jogwar S S., Sustainability and process control: A survey and perspective. Journal of Process Control, 2016, 44: 184-206.
• Christofides P. D., Scattolini R., De La Pena D. M., Liu, J., Distributed model predictive control: A tutorial review and future research directions. Computers & Chemical Engineering, 2013, 51: 21-41.
• Huo X., Karimi H. R., Zhao X., Wang B., Zong G., Adaptive-critic design for decentralized event-triggered control of constrained nonlinear interconnected systems within an identifier-critic framework. IEEE Transactions on Cybernetics, 2021, 52(8): 7478-7491.
• Jiang Q., Yan X., Huang B., Review and perspectives of data-driven distributed monitoring for industrial plant-wide processes. Industrial & Engineering Chemistry Research, 2019, 58(29): 12899-12912.
• Shahab M. A., Mozafari B., Soleymani S., Dehkordi, N. M., Shourkaei, H. M., Guerrero, J. M., Distributed consensus-based fault tolerant control of islanded microgrids. IEEE Transactions on Smart Grid, 2019, 11(1): 37-47.
• Zhou Q., Shahidehpour M., Paaso A., Bahramirad, S., Alabdulwahab, A., Abusorrah, A., Distributed control and communication strategies in networked microgrids. IEEE Communications Surveys & Tutorials, 2020, 22(4): 2586-2633.
• Bemporad A., Barcelli D., Decentralized model predictive control. Networked control systems, 2010: 149-178.
• Stewart B. T., Venkat A. N., Rawlings J. B., Wright, S. J., Pannocchia, G., Cooperative distributed model predictive control. Systems & Control Letters, 2010, 59(8): 460-469.
• Zheng, Y., Song, Y., Hill, D. J., Meng, K., Online distributed MPC-based optimal scheduling for EV charging stations in distribution systems. IEEE transactions on industrial informatics, 2018, 15(2): 638-649.
• Guo, Y., Wu, Q., Gao, H., Chen, X., Østergaard, J., Xin, H., MPC-based coordinated voltage regulation for distribution networks with distributed generation and energy storage system. IEEE Transactions on Sustainable Energy, 2018, 10(4): 1731-1739.
• Zhou, Y., Wang, M., Ahn, S., Distributed model predictive control approach for cooperative car-following with guaranteed local and string stability. Transportation research part B: methodological, 2019, 128: 69-86.
• Sutton R. S., Barto A. G., Reinforcement learning: An introduction. MIT press, 2018
• Zhang K., Yang Z., BaÅŸar T., Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, 2021: 321-384.