(598e) Reduced Space Formulation for Global Optimization with Artificial Neural Networks Embedded

Artificial neural networks (ANNs) are known to be universal approximators [1]. They are utilized in numerous applications for data-driven black-box modeling [2]. In addition, they are likely to gain further importance as digitalization of industry further progresses. In many industrial applications, it is desired to incorporate one or several ANNs into an optimization framework for determining optimal operation strategies or optimal process design [2]. This means that it is desired to embed ANNs in optimization problems. In the previous literature (e.g., [3]), optimization problems with ANNs embedded have mostly been solved by local or stochastic global approaches, e.g., genetic algorithms. However, these methods can fail to identify global optima and give no guarantee of optimality.

Global deterministic optimization of problems with ANNs embedded was done by Smith et al (2013) who used BARON to optimize a flooded bed algae bioreactor that was modeled by an ANN with one hidden layer and three neurons [4]. However, ANNs often comprise numerous neurons in several hidden layers. In these cases, the consideration of ANNs as equality constraints leads to large-scale nonlinear optimization problems. Furthermore, the activation function that is used in each neuron is often a non-linear function, e.g., the hyperbolic tangent function. Currently, the hyperbolic tangent function is not directly available in many global solvers and when represented as summation of terms, its relaxations are relatively weak.

We present a method for deterministic global optimization of optimization problems with ANNs embedded [5], based on recognizing the structure and utilizing it. More specifically, the presented method propagates convex and concave relaxations without any extra variables through ANNs by means of McCormick-based relaxations of algorithms [6,7]. This approach reduces the dimensionality of the optimization problem drastically. In addition, the convex and concave envelopes of nonlinear activation functions are implemented in the presented framework tightening the overall ANN relaxations. The resulting optimization problem is solved using our in-house global deterministic solver MAiNGO [8,9].

The computational performance of the proposed method is illustrated and compared to the solver BARON on four case studies: a test function, a fermentation process, a compressor plant, and a chemical process optimization. Moreover, the scaling of the solution approach with the ANN size is discussed. The results show that the dimensionality of the reduced-space optimization problem is drastically reduced compared to the full-space formulation. Further, the proposed method performs favorable on the case studies compared to BARON, i.e., it shows a speedup.

[1] Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Networks 2(5), 359-366 (1989).

[2] Meireles, M., Almeida, P., Simoes, M.G.: A comprehensive review for industrial applicability of artificial neural networks. IEEE Transactions on Industrial Electronics 50(3), 585-601 (2003).

[3] Nascimento, C.A.O., Giudici, R., Guardani, R.: Neural network based approach for optimization of industrial chemical processes. Computers & Chemical Engineering 24(9-10), 2303-2314 (2000).

[4] Smith, J.D., Neto, A.A., Cremaschi, S., Crunkleton, D.W.: CFD-based optimization of a flooded bed algae bioreactor. Industrial & Engineering Chemistry Research 52(22), 7181-7188 (2013).

[5] Schweidtmann, A.M., Mitsos, A.: Global Deterministic Optimization with ANNs Embedded. Preprint available at arXiv:1801.07114 (2018).

[6] Mitsos, A., Chachuat, B., Barton, P.I.: McCormick-based relaxations of algorithms. SIAM Journal on Optimization 20(2), 573-601 (2009).

[7] McCormick, G.P.: Computability of global solutions to factorable nonconvex programs: Part I - convex underestimating problems. Mathematical Programming 10(1), 147-175 (1976).

[8] Bongartz, D., Najman, J., Scholl, S., Mitsos, A.: MAiNGO: McCormick based Algorithm for mixed integer Nonlinear Global Optimization. Technical report (2018).

[9] Bongartz, D., Mitsos, A.: Deterministic global optimization of process flowsheets in a reduced space using McCormick relaxations. Journal of Global Optimization 20(9), 419 (2017).