(236b) Omlt: Optimization and Machine Learning Toolkit

This talk introduces the optimization and machine learning toolkit (https://github.com/cog-imperial/OMLT, OMLT 1.0), an open-source software package enabling optimization over high-level representations of neural networks (NNs) and gradient-boosted trees (GBTs) [1]. Optimizing over trained surrogate models allows integration of NNs or GBTs into larger decision-making problems. Computer science applications include maximizing a neural acquisition function [2] and verifying neural networks [3]. Engineering applications can utilize black-box optimization or hybridize mechanistic, model-based optimization with surrogate models learned from data [4-6].

OMLT enables engineers and optimizers to easily translate learned machine learning models to optimization formulations. OMLT 1.0 supports GBTs through an ONNX (https://github.com/onnx/onnx) interface and NNs through both ONNX and Keras [7] interfaces. OMLT transforms pre-trained machine learning models into the python-based algebraic modeling language Pyomo [8] to encode optimization formulations. While a few tools exist for specific models [5, 9-10], OMLT is a general tool incorporating both NNs and GBTs, many input models via ONNX, both fully-dense and convolutional layers, several activation functions, and various optimization formulations. The literature often presents different optimization formulations as competitors, e.g. our partition-based formulation competes with the big-M formulation for ReLU NNs [11]. In OMLT, competing optimization formulations become alternatives: users can find the best for a specific application.

We demonstrate applications of OMLT by applying it to case studies including neural network verification, autothermal reformer optimization, and Bayesian optimization. Other examples we envision for OMLT include grey-box optimization [12-13] and other applications relevant to IDEAS [14].

References:

[1] Ceccon F, Jalving J, Haddad J, Thebelt A, Tsay C, Laird CD, Misener R. OMLT: Optimization & Machine Learning Toolkit. arXiv preprint arXiv:2202.02414, 2022.

[2] Volpp M, Fröhlich LP, Fischer K, Doerr A, Falkner S, Hutter F, Daniel C. Meta-learning acquisition functions for transfer learning in bayesian optimization. In Proceedings of The International Conference on Learning Representations (ICLR), 2020.

[3] Botoeva E, Kouvaros P, Kronqvist J, Lomuscio A, Misener R. Efficient verification of ReLU-based neural networks via dependency analysis. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020.

[4] Bhosekar A, Ierapetritou M. Advances in surrogate based modeling, feasibility analysis, and optimization: A review. Computers & Chemical Engineering, 2018.

[5] Schweidtmann AM, Mitsos A. Deterministic global optimization with artificial neural networks embedded. Journal of Optimization Theory and Applications, 2019.

[6] Thebelt A, Wiebe J, Kronqvist J, Tsay C, Misener R. Maximizing information from chemical engineering data sets: Applications to machine learning Chemical Engineering Science, 2022.

[7] Chollet F et al. Keras. https://keras.io, 2015.

[8] Bynum ML, Hackebeil GA, Hart WE, Laird CD, Nicholson BL, Siirola JD, Watson JP, Woodruff DL. Pyomo-optimization modeling in Python. Springer, 2021.

[9] Lueg L, Grimstad B, Mitsos A, Schweidtmann AM. reluMIP: Open source tool for MILP optimization of ReLU neural networks, 2021. URL https: //github.com/ChemEngAI/ReLU_ANN_MILP.

[10] Maragno D, Wiberg H, Bertsimas D, Birbil SI, den Hertog D, Fajemisin A. Mixed-integer optimization with constraint learning. arXiv preprint arXiv:2111.04469, 2021.

[11] Tsay C, Kronqvist J, Thebelt A, Misener R. Partition-based formulations for mixed-integer optimization of trained ReLU neural networks. Advances in Neural Information Processing Systems, 2021.

[12] Boukouvala F, Hasan MM, Floudas CA. Global optimization of general constrained grey-box models: new method and its application to constrained PDEs for pressure swing adsorption. Journal of Global Optimization, 2017.

[13] Wilson ZT, Sahinidis NV. The ALAMO approach to machine learning. Computers & Chemical Engineering, 2017.

[14] Lee A, Ghouse JH, Eslick JC, Laird CD, Siirola JD, Zamarripa MA, Gunter D, Shinn JH, Dowling AW, Bhattacharyya D, Biegler LT. The IDAES process modeling framework and model library—Flexibility for process simulation and optimization. Journal of Advanced Manufacturing and Processing, 2021.