(472e) On Testing Methods for Image-Based Control Systems for Next-Generation Manufacturing

Next-generation manufacturing utilizes process automation and data to improve system performance. This can include new data modalities such as image-based sensors with image processing algorithms integrated with typical process automation concepts, which can enable controllers based on camera sensors to measure process variables that would otherwise be impractical to be collected online (e.g., cell counting which is traditionally measured offline). These image-based control (IBC) systems have been used in applications that range from bioprocess monitoring [1] and drug discovery [2] to robot manipulators [3] and autonomous driving systems [4]. Although significant advances in visual feedback control using image-based sensors and image processing algorithms have been reported in the literature [5][6][7], these control architectures integrated with a vision system may be potential targets of cyberattacks that aim to falsify image-based measurements provided to the controller. In light of this, simulation-based studies involving image data modality are needed to understand the impacts of cyberattacks on image-based control systems and propose control/detection strategies to detect and prevent malicious actions.

To enhance the performance of autonomous systems, different IBC systems have been proposed using a variety of monitoring systems and control architectures, including classical proportional-integral-derivative (PID) controllers [3] and advanced control formulations (e.g., model predictive control (MPC) [6]). In [3], an hexarotor image-based visual servo control has been proposed using a neural network-based PID controller integrated with a visual system, where physical data was collected and simulations were performed in MATLAB via the Robotics Toolbox [8]. In [6], an image-based control design using a nonlinear MPC has been developed as a strategy for an unmanned aerial vehicle, where the constraints related to actuator and visibility limitations can be added to the MPC formulation. For applications in the chemical process industry, [9] describes a process monitoring system for an industrial boiler based on a real-time image analysis framework. In [10], a real-time image analysis system has been used to estimate bubble sizes at a phosphorus oxide flotation process. In [11], bubbles in fluids have been simulated as a 3D animation with physical realism by considering the physics of dissolved gas diffusion, nucleation, and their interaction with liquids.

Moving towards realistic simulations of IBC systems, 3D programming software such as Blender, which has an embedded Python interpreter, may help with creating and testing IBC systems for different industrial applications before being implemented on the real system, which can offer a manner to test how new next-generation manufacturing strategies such as image-based control schemes might behave in the actual process. Motivated by the advances in visual feedback control systems and animation, we showcase how Blender can be used to develop and test IBC designs under different control strategies, and evaluate the implications of different cyberattack events involving image data tampering on process safety. Simulations of level control in a tank are used to illustrate the proposed methodology. The level in the tank is simulated in the Blender Python programming interface, and this information is used to adjust the position of the height of the tank (modeled as the top of a plane) in Blender. Then, a render of the image is taken and subsequently processed via Pillow [12] to enable the pixel corresponding to the top of the tank to be identified. This is then used to estimate the height of the tank, which is fed to a proportional-integral (PI) controller to control the process. The process is simulated in the absence of a cyberattack, and also the presence of attacks, to demonstrate how Blender can be used to provide evaluations of image-based control laws under different conditions. Finally, we discuss how Blender can also be used to add noise to an image to give it a characteristic that might be a part of other cyberattack detection algorithms [13].

References:

1. Bluma, Arne, Tim Höpfner, Patrick Lindner, Christoph Rehbok, Sascha Beutel, Daniel Riechers, Bernd Hitzmann, and Thomas Scheper. "In-situ imaging sensors for bioprocess monitoring: state of the art." Analytical and Bioanalytical Chemistry 398.6 (2010): 2429-2438.
2. Chandrasekaran, Srinivas Niranj, Hugo Ceulemans, Justin D. Boyd, and Anne E. Carpenter. "Image-based profiling for drug discovery: due for a machine-learning upgrade?." Nature Reviews Drug Discovery 20.2 (2021): 145-159.
3. Su, Yuxin, and Chunhong Zheng. "A simple PID control for asymptotic visual regulation of robot manipulators." International Journal of Robust and Nonlinear Control 21.13 (2011): 1525-1540.
4. Bengler, Klaus, Klaus Dietmayer, Berthold Färber, Markus Maurer, Christoph Stiller, and Hermann Winner. "Three decades of driver assistance systems: Review and future perspectives." IEEE Intelligent Transportation Systems Magazine 6.4 (2014): 6-22.
5. Lopez-Franco, Carlos, Javier Gomez-Avila, Alma Y. Alanis, Nancy Arana-Daniel, and Carlos Villaseñor. "Visual servoing for an autonomous hexarotor using a neural network based PID controller." Sensors 17.8 (2017): 1865.
6. Lee, Daewon, Hyon Lim, and H. Jin Kim. "Obstacle avoidance using image-based visual servoing integrated with nonlinear model predictive control." 2011 50th IEEE Conference on Decision and Control and European Control Conference. IEEE, 2011.
7. Hajiloo, Amir, Mohammad Keshmiri, Wen-Fang Xie, and Ting-Ting Wang. "Robust online model predictive control for a constrained image-based visual servoing." IEEE Transactions on Industrial Electronics 63.4 (2015): 2242-2250.
8. Corke, Peter I. "A robotics toolbox for MATLAB." IEEE Robotics & Automation Magazine 3.1 (1996): 24-32.
9. Yu, Honglu, and John F. MacGregor. "Monitoring flames in an industrial boiler using multivariate image analysis." AIChE Journal 50.7 (2004): 1474-1483.
10. Lin, Bao, Bodil Recke, Jørgen K. H. Knudsen, and Sten Bay Jørgensen. "Bubble size estimation for flotation processes." Minerals Engineering 21.7 (2008): 539-548.
11. Cleary, Paul W., Soon Hyoung Pyo, Mahesh Prakash, and Bon Ki Koo, "Bubbling and frothing liquids." ACM SIGGRAPH 2007 papers. 2007. 97-es.

12.Clark, A. (2015). Pillow (PIL Fork) Documentation. readthedocs. Retrieved from https://buildmedia.readthedocs.org/media/pdf/pillow/latest/pillow.pdf

13. H. Farid, “Digital image forensics: lecture notes, exercise, and matlab code for a survey course in digital image and video forensics,” p. 199, 2012, accessed: 2021-12-10.