(172e) Interpretability for Moving Toward Verification of Advanced and Data-Driven Control

There is an increasing emphasis on the capabilities of the use of automation for handling a variety of tasks that involve complex decision-making or learning from past data. Though this presents many opportunities for systems which are able to adapt in the face of complicated or unforeseen circumstances, it also poses many challenges in terms of verification at the design and test stage that such systems will work as expected in practice. Prior work in our group explored the development of constraints for an advanced optimization-based control design known as economic model predictive control (EMPC) [1] that could be adjusted via learning over time within a range in which it was guaranteed that they would not impact closed-loop stability [2]. However, it would be expected that there would be many circumstances in which restricting adaptability of control systems only to constraints which do not impact control-theoretic guarantees may not be practical. In such a case, it is necessary to develop strategies for gaining trust for learned control actions even when guarantees cannot be made. It has been suggested that one way in which humans may learn to trust artificially intelligent systems is by causing their decision-making process to be interpretable [3]. However, while interpretability for machine learning techniques such as neural networks has received a good deal of attention (e.g., [4]), there is not currently a broadly-accepted means for incorporating interpretability in control design, particularly in cases where the controller may make non-intuitive decisions by solving an optimization problem to determine its control actions, or where a model-based controller may incorporate a model or constraints that adapt over time to changing environments.

Motivated by the above considerations, this work provides an investigation of how interpretability may be incorporated within the context of optimization-based control, and particularly within the context of EMPC with both time-invariant first-principles models (but where it may still compute non-intuitive control actions to optimize profits), and when it utilizes a neural network model for the process that is updated using process data as the process conditions change over time. The initial focus will be on defining cases in which the actions of the EMPC would not be interpretable, and on defining interpretability in a variety of mathematical contexts (e.g., those based on pattern recognition or directionality in process behavior when the input trajectories take certain directions). It will be analyzed how the EMPC formulation may be modified to cause it to be more interpretable with respect to these different metrics. Subsequently, we will investigate how approaches to neural network interpretability from the literature (e.g., approaches which analyze which nodes are being activated for certain inputs to the neural network, and how sensitive the outputs of the network are to changes in certain weights [3,5]) can be used to seek to understand how the changes in the control actions over time as the process model changes might be made understandable to a human in light of the model modifications and the process data so that an operator could assess whether these actions are appropriate or not.

[1] M. Ellis, H. Durand, and P. D. Christofides. “A tutorial review of economic model predictive control methods.” Journal of Process Control, 24:1156–1178, 2014.

[2] H. Durand. “Responsive economic model predictive control for next-generation manufacturing.” Mathematics, 8:259, 38 pages, 2020.

[3] S. Chakraborty, R. Tomsett, R. Raghavendra, D. Harborne, M. Alzantot, F. Cerutti, M. Srivastava, A. Preece, S. Julier, R. M. Rao, T. D. Kelley, D. Braines, M. Sensoy, C. J. Willis, and P. Gurram. “Interpretability of deep learning models: A survey of results.” In Proceedings of the IEEE Smart World Congress, San Francisco, CA, 2017.

[4] Zhang, Q., Y. N. Wu, and S.-C. Zhu. “Interpretable convolutional neural networks.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 8827-8836, 2018.

[5] Gilpin, L. H., D. Bau, B. Z. Yuan, A. Bajwa, M. Specter and L. Kagal. “Explaining explanations: An overview of interpretability of machine learning.” In Proceedings of the 2018 IEEE 5^th International Conference on Data Science and Advanced Analytics, Turin, Italy, 80-89, 2018.