(372c) Process Design Automation Using Reinforcement Learning

Today process design studies have been done in the manner of expressing possible designs in a superstructure and solving a nonconvex optimization problem. Such a framework solves the resulting mixed integer nonlinear program for a global or local optimum [1]. They are generally focused on getting a solution fast using derivative information. These are impractical when the derivatives are not evaluated analytically as in the case where the process contains a black-box model unit, or if the derivatives are not numerically reliable, which is often the case when the process is extremely large and exhibits severe nonlinearity [2]. Furthermore, since they generally do not utilize the pattern from sampled data, the approach becomes intractable if the simulation is computationally expensive. These are not suitable for a recent trend which integrates external modules such as computational fluid dynamic (CFD) models for high-fidelity simulation [3].

Under the recognition of these problems, several methodologies to overcome the issues have been suggested. They include obtaining derivatives from a surrogate model, or not using derivative at all by utilizing the evolutionary algorithm [3,4]. They use the pattern of solutions from the sampling and enable to obtain the solution efficiently although it may not be exact. All these methods are relatively slow and, in many cases, do not guarantee convergence to global solutions, and generally hybridization with existing algorithms are preferred.

Nonetheless, methodologies mentioned earlier are all focused on solving a given design problem. If a new design problem is considered, the formulation must be repeated again, and the knowledge experienced during the solution procedure cannot be used to solve other problems. For example, if the feed composition or the parameters in reaction kinetics change, they need to formulate the problem and solve again.

This study presents a novel methodology of using reinforcement learning for a machine to perform diverse process design tasks and to master process design itself. We first establish an environment defined only with unit operations. We then develop an algorithm which converts the representation of a given process from a direct graph to a canonically equivalent tensor. We configure neural networks that take a process tensor as an input and output the desired type and specifications of a unit in the form of a probability distribution. Agent performing process design repeats the sequence of observing process tensor in each step, performing an action from the neural networks, and getting the profit of the process as a reward from the simulation result.

In numerous off-line learning processes, we could make the machine to go through diverse process design tasks by changing the feed composition and the parameters in reaction kinetics. We were able to confirm that the machine can obtain reliable solution not only from the tasks for which it was trained but also from the similar tasks that were not given during the training phase. Furthermore, we found that the more the machine learns, the shorter the time it took to design a process.

[1] Chen, Q., & Grossmann, I. E. (2017). Recent Developments and Challenges in Optimization-Based Process Synthesis. Annual Review of Chemical and Biomolecular Engineering, 8(1), 249–283. https://doi.org/10.1146/annurev-chembioeng-080615-033546

[2] Zhou, T., Zhou, Y., & Sundmacher, K. (2017). A hybrid stochastic–deterministic optimization approach for integrated solvent and process design. Chemical Engineering Science, 159, 207–216. https://doi.org/10.1016/j.ces.2016.03.01

[3] Boukouvala, F., Misener, R., & Floudas, C. A. (2016). Global optimization advances in Mixed-Integer Nonlinear Programming, MINLP, and Constrained Derivative-Free Optimization, CDFO. European Journal of Operational Research, 252(3), 701–727. https://doi.org/10.1016/j.ejor.2015.12.018

[4] Neveux, T. (2018). Ab-initio process synthesis using evolutionary programming. Chemical Engineering Science, 185, 209–221. https://doi.org/10.1016/j.ces.2018.04.015