(286e) Image Prediction for Model Predictive Control

Autonomous robots equipped with vision and sensing have recently come to be implemented in chemical plants for process monitoring and inspection (such as the ANYmal robot [2]), with the expectation that human-robot interaction will be commonplace in next-generation manufacturing plants [7]. Machine learning based approaches are used extensively to help robots achieve tasks, such as movement [5], path planning [6], object detection [1], and more. While generally favorable in terms of computation times, machine learning models require a “black box” process model lacking insights necessary to interpret why actions are being chosen, which may be especially detrimental in contexts where flawless operation is crucial for safety or the completion of mission objectives (e.g. inspection of a plant after a disaster where hazardous materials may be present) [4]. Object detection in particular is a critical aspect (YOLOv4 [1] is an example); machine learning models are trained to detect and classify objects with a confidence interval in a robot’s field of vision, which may be particularly useful for inspecting places where it is difficult for humans to go. However, due to the black box model and confidence interval, it may be beneficial to supplement object detection with an image-based control methodology utilizing predictive control that can steer the robot in ways which could increase its confidence in its assessment of its environment to promote safe and accurate performance.

In this talk, we discuss a concept for image prediction for integration with predictive control to investigate how and whether predictive control algorithms could be useful in a chemical process context if they seek to select movements for an agent that would aid it in distinguishing between different possible scenarios with respect to its environment. The premise of these studies is that an agent (e.g., robot) has developed several alternative possibilities of what it considers its environment might look like, but cannot with the available data fully come to a conclusion on what its environment looks like. It needs to gather new data to determine this; however, we would like it to achieve a control objective while gathering this new data (e.g., to gather this data in a manner that allows the agent to come to a conclusion about which of the possibilities is accurate as quickly as possible). We consider that there are directions that the robot could move that could enable it to figure out this information quickly, and others where the different options for what the environment looks like would continue to not be as readily distinguishable. We would like to design a strategy by which the robot takes a path by which it may most readily determine which of the visual models of its environment is the accurate one.

The first step in moving toward this goal is to select a simulation framework for images and analyze cases where, if a three-dimensional environment was to form the process model in a predictive controller, different movements of the agent around these environments would demonstrate different visual cues that could be used to select how to maneuver the robot to achieve the desired differentiation between potential environments. For this, we use the OpenGL specification for rendering graphics of single cubes with different color gradients to show differences visually between trajectories which an agent could take around an object and to aid with considering how a control design that takes these types of images as models might be formulated [3]. We discuss how the character of the image data should inform the design of the objective functions, modeling strategies, and constraints in a model predictive control formulation that makes predictions of future images to aid in navigation. We also discuss what is gained and lost if full image prediction is not performed when trying to sort out the environment description.

[1] Bochkovskiy, A., C.-Y. Wang, and H.-Y. M. Liao. "Yolov4: Optimal speed and accuracy of object detection." arXiv preprint arXiv:2004.10934 (2020).

[2] Hutter, M., C. Gehring, A. Lauber, F. Gunther, C. D. Bellicoso, V. Tsounis, P. Frankhauser, R. Diethelm, S. Bachmann, M. Bloesch, H. Kolvenbach, M. Bjelonic, L. Isler, and K. Meyer. "Anymal-toward legged robots for harsh environments." Advanced Robotics 31.17 (2017): 918-931.

[3] Oyama, H., D. Messina, R. O'Neill, S. Cherney, M. Rahman, K. K. Rangan, G. Gjonaj, and H. Durand, "Test Methods for Image-Based Information in Next-Generation Manufacturing," Proceedings of the IFAC Symposium on Dynamics and Control of Process Systems, including Biosystems (DYCOPS), Busan, Republic of Korea, in press.

[4] Pereira, A., and C. Thomas. "Challenges of machine learning applied to safety-critical cyber-physical systems." Machine Learning and Knowledge Extraction 2.4 (2020): 579-602.

[5] Tan, J., T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. "Sim-to-real: Learning agile locomotion for quadruped robots." arXiv preprint arXiv:1804.10332 (2018).

[6] Tullu, A., B. Endale, A. Wondosen, and H.-Y. Hwang. "Machine learning approach to real-time 3D path planning for autonomous navigation of unmanned aerial vehicle." Applied Sciences 11.10 (2021): 4706.

[7] Yu, L., E. Yang, P. Ren, C. Luo, G. Dobie, D. Gu, and X. Yan. "Inspection robots in oil and gas industry: a review of current solutions and future trends." 2019 25th International Conference on Automation and Computing (ICAC). IEEE (2019).