(373o) Multi-Fidelity Reinforcement Learning for Dynamic Optimisation of Laminar Mixing System

Plug flow is an optimal flow pattern characterised by minimal mixing along the direction of flow and efficient cross-mixing, making it highly desirable for various applications such as food processing and environmental engineering. However, the laminar flow nature in the majority of these applications can adversely impact the mixing characteristics of plug flow. Some studies have demonstrated that introducing pulsating oscillatory inlet flow can enhance mixing efficiency of plug flow for laminar flow applications. Therefore, optimising the parameters of the pulsating flow can be of paramount importance for better control over the flow dynamics of plug flow and, consequently, residence time distribution, and product quality.

Reinforcement Learning (RL), a Machine Learning (ML) paradigm, focuses on training agents to make sequential optimal decisions by interacting with an environment. RL has the potential to play a significant role in Computational Fluid Dynamics (CFD)-based optimisation by facilitating optimal decision. However, the combination of RL with CFD presents a formidable challenge mainly stemming from the inherent computational expense of CFD simulations and the data-hungry nature of RL algorithms.

To address this challenge, we propose a two-fold approach. Firstly, we leverage ML techniques to develop lightweight emulators tailored to simulate our CFD-based environments closely. These emulators act as surrogate environments, closely mimicking the behaviour of CFD solvers, thereby enabling RL algorithms to efficiently interact and collect training data. By employing these emulators, we mitigate the computational burden and enhance the speed of data generation for RL training via accelerating the data generation process for RL training. This strategy not only enhances the speed of learning but also facilitates scalability, making it a practical and efficient method for training RL algorithms in complex fluid dynamics scenarios.

Secondly, we refine the training procedure of RL algorithms by introducing interaction with multi-fidelity CFD simulators. The multi-fidelity CFD simulators act as a hierarchy of tasks, with each fidelity level representing a different level of complexity or detail in the simulation. Through its interactions with these emulators, the RL agent can extract and generalise underlying patterns and strategies. This learned knowledge becomes a transferable asset, enhancing the agent's adaptability and reducing the amount of training required when confronted with novel, high-fidelity CFD simulations. Transfer learning in the context of multi-fidelity simulations allows the RL agent to build a foundation of understanding in simpler environments and gradually refine its decision-making capabilities as it encounters more complex scenarios where the complexity may arise from the flow dynamics and/or flow geometry. This approach not only accelerates the learning process but also enhances the agent's ability to generalise its knowledge to previously unseen, challenging situations.

In this research, we seek to achieve two main objectives: (1) to formulate optimisation of a simulated laminar mixing system as a dynamic autonomous RL problem; and (2) to analyse specific mixing flow characteristics discovered by an optimal policy, which intends to dynamically adjust inlet conditions based on received observation data.