(376j) Assessing Team Performance in Control Rooms: Integrating Eye Tracking with Multimodal Data Analysis for Insights

The Control Room is like the brain of the system, where multiple operators play their role as a team to ensure the plant operates safely and efficiently in both normal and emergencies. As automation and AI tools become increasingly prevalent, operators face handling a large volume of data (Naef and Lefsrud, 2023). This growing data provides opportunities for deeper insights. However, it simultaneously poses challenges, including the risk of overwhelming operators with information overload, which can affect their judgment and decision-making capabilities. In such situations, effective communication and collaboration among operators become crucial for maintaining situational awareness and making informed decisions (Kluge et al., 2014). Shared Mental Model, defined as “organized knowledge shared among team members” by Orasanu & Salas(1993), plays a crucial role in fostering common understanding and interpretation of tasks, leading to improved teamwork, coordination, and communication (Kluge et al., 2014). Teams with a strong Shared Mental Model demonstrate faster reactions, greater accuracy, and higher mission success rates, while also enhancing their capacity to handle novel tasks and situations (Espevik et al., 2011). This collective understanding and alignment among team members contribute significantly to overall team performance and effectiveness.

To evaluate Shared Mental Model, it's crucial to delve into the cognitive behaviors of team members and understand their situational awareness. Recent technological advancements have enabled physiological measures like eye movement to study cognitive behaviors of the operators in modern Control Rooms. For instance, Salehi et al. (2018) found that novice operators generally focused less (as evidenced by shorter fixation durations) on pivotal HMI (Human-Machine Interface) areas compared to their expert counterparts. In our previous work, we have delved deep into cognitive processes and mental models using eye tracking technology. Our findings reveal that operators familiar with process dynamics focused primarily on essential variables pertinent to the current situation (Bhavsar et al., 2017).

Additionally, we developed a novel method using eye tracking to measure operators' attention in control rooms with multiple displays, including Large Screen Displays (LSD) for shared situational awareness. Alongside cognitive aspects, assessing team communication is integral to understanding Shared Mental Model. Study by Kluge et al. (2014) revealed that teams with Shared Mental Models engaged in less explicit communication, relied more on information transfer. Another study by Khawaja et al. (2012) showed that teams exhibited decreased agreement and increased disagreement during challenging tasks due to high cognitive load. Given these insights, there is a pressing need to devise a comprehensive methodology that utilizes multimodal data sources, such as operator attention, interaction patterns, and actions, to gain a holistic understanding of operator behaviors conducive to efficient teamwork.

In this work, we proposed a novel methodology for evaluating team performance in collaborative process control rooms, with a particular focus on environments featuring Large Screen Displays (LSD) positioned separately from individual workstations, a unique aspect not previously explored. Our approach utilizes eye-tracking technology to measure attention within the control room setup. Additionally, we incorporate multimodal data sources, including voice and text messages, to analyze team interactions, along with process information such as operator actions, alarm responses, and task completion times, providing further insights. The proposed method is validated on an in-house test bed replicating the environment of a collaborative control room with multiple displays, including one LSD tailored for collaborative monitoring. Human subjects, acting as operators in an Ethanol production unit simulator control room, participated in the validation process. The results highlighted the effectiveness of our methodology in capturing team performance in modern collaborative control room setups. The collected data were analyzed to measure the situational awareness, important information shared, and finally, action taken to measure team performance, shedding light on shared attention patterns, team dynamics, and collaborative decision-making processes. These insights are pivotal for enhancing individual and team performance and understanding Shared Mental Models across various industrial settings.

References:

Naef, M., & Lefsrud, L. (2023). Smooth operator: Aligning performance assessment methods with design and operating objectives. Journal of Loss Prevention in the Process Industries, 105158.

Kluge, A., Nazir, S., & Manca, D. (2014). Advanced applications in process control and training needs of field and control room operators. IIE Transactions on Occupational Ergonomics and Human Factors, 2(3-4), 121-136.

Orasanu, J., & Salas, E. (1993). Team decision making in complex environments.

Espevik, R., Johnsen, B. H., & Eid, J. (2011). Communication and performance in co-located and distributed teams: an issue of shared mental models of team members?. Military Psychology, 23(6), 616-638.

Salehi, S., Kiran, R., Jeon, J., Kang, Z., Cokely, E. T., & Ybarra, V. (2018). Developing a cross-disciplinary, scenario-based training approach integrated with eye tracking data collection to enhance situational awareness in offshore oil and gas operations. Journal of Loss Prevention in the Process Industries, 56, 78-94.

Bhavsar, P., Srinivasan, B., & Srinivasan, R. (2017). Quantifying situation awareness of control room operators using eye-gaze behavior. Computers & chemical engineering, 106, 191-201.

Khawaja, M. A., Chen, F., & Marcus, N. (2012). Analysis of collaborative communication for linguistic cues of cognitive load. Human factors, 54(4), 518-529.