(268c) Fusion of Eye-Tracking and Electroencephalography (EEG) Based Metrics for Estimating the Cognitive Workload of Control Room Operators in Process Industries

Process industries are highly hazardous and accidents keep occurring in these industries. Thus, it is imperative to develop resilient safety barriers. While there are several layers of protection that prevent the hazards leading to accidents, human form on significant layer. This can be ascertained by the fact that majority of accidents in process industries occur because of human errors; human errors account for nearly 60-80% of process industry accidents (Brauer, 2016). A detailed analysis of severe industrial accidents that occurred in the period spanning 2008 to 2018 revealed that about 76.1% of these accidents can trace back their root cause to human failures (Jung et al., 2020). The advent of rapid digitization and concomitant automation has changed the nature of human-machine interaction, and hence the role of humans, in process industries. The nature of the operator’s task has shifted from the one involving emphasis on perceptual-motor skills to the one involving cognitive activities such as monitoring, diagnosis, prognosis, decision making, and problem-solving (Pascual et al., 2019). These cognitive activities need to be carried out efficiently and correctly by an operator to achieve the design goals and deal with process abnormalities that are outside the outside the purview of automation systems. Inefficiencies in human cognitive performance often translate to high cognitive workload that can have catastrophic consequences in terms of compromising the safety in process industries. Cognitive workload is one of the important constructs of human performance and an increase in it leads to degradation in performance (Ghalenoei et al., 2022). This work addresses the critical need to estimate cognitive workload of process industry control room operators using a fusion of eye-tracking and electroencephalography (EEG) derived features.

Despite advances in human performance assessment, traditional methods often overlook critical cognitive aspects of operators' performance. Evaluation is typically based on metrics like success rates, deviations from standard operating procedures, and response times (Lee et al., 2000), which fail to fully capture the underlying reasons behind sub-optimal performance. Such underlying reasons may include cognitive workload, shortcomings in mental models and design of HMI. Advancement in sensor technology has made it possible for researchers to tap on physiological measurements for getting insights into the cognitive aspects of human performance, including cognitive workload. Physiological parameters reflect intrinsic mental state of an individual and provide critical insights into those aspects of cognitive behavior which are otherwise not directly observable such as level of attention, information acquisition pattern, thinking strategy, and workload. A variety of sensors have been developed to measure various physiological parameters (Zheng et al., 2014) and many of these are minimally invasive. These sensors primarily include eye tracking, and electroencephalography (EEG). Eye tracking provides critical insights into visual behavior. For instance, expert nuclear power plant operators have a different eye gaze pattern during stressful emergency accident situations, marked by higher total fixation durations compared to novices (Liu et al., 2019) thus providing critical insights into information acquisition pattern. Likewise, electroencephalography (EEG) provides critical information about operators’ mental states, such as levels of cognitive workload (Liang et al., 2018). For instance, spectral power of EEG theta has been found to be sensitive to mental workload of air traffic controllers (Dasari et al., 2017). However, depending on a single sensor may not often be adequate in a real setting where factors such as measurement noise, and artifacts induced by a subject’s movement can affect the accuracy and robustness of single sensor-based metrics. Moreover, some aspects of performance may not be sensitive to the metrics obtained from one sensor while the other sensor-based metrics can capture the same (Debie et al., 2021). Therefore, utilizing features from both sensors is beneficial. For instance, a classification accuracy of 93% for visual hazard detection was attained when features from eye-tracking and EEG were fused compared to accuracies of 73% (for eye-tracking) and 83% (for EEG) obtained independently (Noghabaei et al., 2021). Our eye-tracking research (Sharma et al., 2016; Bhavsar et al., 2017) and EEG studies (Iqbal et al., 2020, 2021) provide valuable insights into the cognitive behavior of control room operators. Experts have lower gaze entropy than novices, indicating more directed attention (Bhavsar et al., 2017). EEG can identify mismatches between an operator's mental models and actual process behavior (Iqbal et al., 2020).

In the current work, we study the added benefit of the fusion of eye-tracking and EEG for operator performance assessment. Ten participants were involved in the study, and each of them carried out several repetitive trials. During a typical trial, a participant has to carry out six tasks (each corresponding to a disturbance scenario). Overall, these participants performed 81 trials resulting in 438 task-participant pairs. During these tasks, participants' brain activity and eye gaze are recorded via a single electrode EEG sensor and Tobii 300 eye tracker, respectively. In addition, process data, alarm information, and operator actions are also recorded. We extracted EEG and eye tracking based metrics for each of the tasks to populate a feature matrix. These metrics include power spectral densities of pupil diameter in the range of 0-2 Hz, average pupil diameter, fixation duration-based (gaze) metrics and power spectral densities of EEG in the range of 1-30 Hz. We trained a decision tree-based model, based on several combinations of features, for estimating cognitive workload of operators. Our results indicate that fusion of metrics results in a marked improvement in assessing the cognitive workload of control room operators compared to using features from EEG or eye tracking alone. For instance, classification accuracy of the decision tree-based model increased from a minimum of 45% (using EEG based features) to a maximum of around 69% using a fusion of metrics obtained from EEG, pupil, and gaze data. The improvement in results is hypothesized to be due to differences in dynamics of EEG and eye data. We also developed an operator independent generic model for estimating cognitive workload during a task which achieved an accuracy of about 67%. The work has the potential to assess cognitive aspects of performance, and evaluate expertise during training, thus providing a comprehensive and robust performance assessment. This will help ensure safety and quality of product in process industries. In a broader perspective, it can guide the application of human factors for improving safety in industries by providing objective information about the cognitive aspects of performance.

Keywords: Process safety, Operator performance, Eye Tracking, Electroencephalography (EEG), Decision Trees

Acknowledgement: This work is partially funded by American Express Lab for Data Analytics,

Risk and Technology, Indian Institute of Technology Madras

References

Ghalenoei, M., Mortazavi, S. B., Mazloumi, A., & Pakpour, A. H. (2022). Impact of workload on cognitive performance of control room operators. Cognition, Technology & Work, 24(1), 195-207.

Bhavsar, P., Srinivasan, B., & Srinivasan, R. (2017). Quantifying situation awareness of control room operators using eye-gaze behavior. Computers & chemical engineering, 106, 191-201.

Brauer, R. L. (2016). Safety and health for engineers. John Wiley & Sons.

Diego-Mas, J. A., Garzon-Leal, D., Poveda-Bautista, R., & Alcaide-Marzal, J. (2019). User-interfaces layout optimization using eye-tracking, mouse movements and genetic algorithms. Applied ergonomics, 78, 197-209.

Iqbal, M. U., Shahab, M. A., Choudhary, M., Srinivasan, B., & Srinivasan, R. (2021). Electroencephalography (EEG) based cognitive measures for evaluating the effectiveness of operator training. Process Safety and Environmental Protection, 150, 51-67.

Iqbal, M. U., Srinivasan, B., & Srinivasan, R. (2020). Dynamic assessment of control room operator’s cognitive workload using Electroencephalography (EEG). Computers & Chemical Engineering, 141, 106726.

Jung, S., Woo, J., & Kang, C. (2020). Analysis of severe industrial accidents caused by hazardous chemicals in South Korea from January 2008 to June 2018. Saf. Sci. 124, 104580

Lee, S., Jeong, I., & Il, M. (2000). Development of evaluation algorithms for operator training system. Computers & Chemical Engineering, 24(2-7), 1517-1522.

Liang, Y., Liang, W., Qu, J., & Yang, J. (2018, October). Experimental study on EEG with different cognitive load. In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 4351-4356). IEEE.

Noghabaei, M., Han, K., & Albert, A. (2021). Feasibility Study to Identify Brain Activity and Eye-Tracking Features for Assessing Hazard Recognition Using Consumer-Grade Wearables in an Immersive Virtual Environment. Journal of Construction Engineering and Management, 147(9), 04021104.

Porthin, M., Liinasuo, M., & Kling, T. (2020). Effects of digitalization of nuclear power plant control rooms on human reliability analysis–A review. Reliability Engineering & System Safety, 194, 106415.

Shahab, M. A., Iqbal, M. U., Srinivasan, B., & Srinivasan, R. (2022). HMM-based models of control room operator’s cognition during process abnormalities. 1. formalism and model identification. Journal of Loss Prevention in Process Industries, 104748.

Sharma, C., Bhavsar, P., Srinivasan, B., & Srinivasan, R. (2016). Eye gaze movement studies of control room operators: A novel approach to improve process safety. Computers & Chemical Engineering, 85, 43-57.