(376i) Investigating Students’ Experience with Generative Chatbots in Chemical Engineering Education

Artificial Intelligence (AI), mainly thanks to the rise of Large Language Models, has recently created much traction, forcing educators worldwide to reflect upon its potential use (and misuse) if used in coursework. These systems have gained considerable interest in educational research and practice due to their ability to learn from and generate natural language. Thanks to the extensive pre-training these models are exposed to, they obtain knowledge about a wide range of topics and are suitable for application in many different contexts and fields. This ability to extract content and critical information from text offers a powerful tool for enhancing the learning experience. According to Huang et al. (2022), these AI tools herald a new era in education, offering unparalleled potential to revolutionize learning experiences and outcomes. They could be used to identify knowledge gaps, aid formative assessment, and provide personalized feedback (Hopfenbeck, 2023; Dai et al., 2023). Researchers have also been evaluating how LLMs perform in a real educational context. Hicke et al. (2023) assess the efficacy of LLMs in generating accurate teacher responses. Xiao et al. (2023) apply LLMs to real-world classroom settings, implementing a reading comprehension exercise generation system that provides students with high-quality and personalized reading materials.

This work focuses on intelligent tutoring systems in a real educational context. We introduced a software tool called ChatGMP, an LLM trained in Good Manufacturing Practice (GMP) data, to a course taught at the Technical University of Denmark (DTU). In this course, students are required to participate in an audit exercise, where they pretend to be an auditor inquiring a company, represented by the teachers, about their good manufacturing practices. During the audit, students can ask questions regarding available documents and practices adopted by the company, which are indispensable to evaluate whether the company in question could be a valuable business partner or if the non-conformities found are too severe.

This year, we started investigating the replacement of the physical teachers in this exercise with a generative AI model, called ChatGMP. The model is a pre-trained Transformer (Vaswani et al., 2017), FLAN-T5 (Chung et al., 2022), used to perform a question-answering task. The answers are retrieved through prompt engineering, where semantic search is performed over the historical database of question-answer pairs. To enable the students to interact with the intelligent tutoring system, which was presented in the form of a chatbot, we built an interactive GUI. ChatGMP was tested in three audits out of twenty-one total groups. The model managed to complete the three audits, answering the students’ questions in a meaningful and complete manner and showing all the requested documents. A post-audit survey shows that all three groups participating in the audit exercise were happy about the experience and were satisfied with the quality of the responses. This marks the first step in successfully automating this task, which would enable more students to attend the course, since currently the yearly uptake is limited by the extensive time required by this exercise.

References

Huang, K. F. Hew, L. K. Fryer. (2022). Chatbots for language learning—are they really useful? a systematic review of chatbot-supported language learning, Journal of Computer Assisted Learning 38. 237–257.

T. N. Hopfenbeck. (2023). The future of educational assessment: self-assessment, grit and chatgtp?

W. Dai, J. Lin, F. Jin, T. Li, Y.-S. Tsai, D. Gasevic, G. Chen. (2023). Can large language models provide feedback to students? a case study on chatgpt.

Y. Hicke, A. Masand, W. Guo, T. Gangavarapu. (2023). Assessing the efficacy of large language models in generating accurate teacher responses, in Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Association for Computational Linguistics, Toronto, Canada, pp. 745–755. URL: https://aclant hology.org/2023.bea-1.60. doi:10.18653/v1/2023.bea-1.60.

Xiao, S. X. Xu, K. Zhang, Y. Wang, L. Xia. (2023). Evaluating reading comprehension exercises generated by LLMs: A showcase of ChatGPT in education applications, in Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Association for Computational Linguistics, Toronto, Canada, pp. 610–625. URL: https://aclanthology.org/2023.bea- 1.52. doi:10.18653/v1/2023.bea-1.52.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.

Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., ... & Wei, J. (2022). Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.