(664b) Developing a Framework for Enhanced Troubleshooting with Large Language Models

Large language models (LLM’s) have received significant attention in recent months with the interest in the capabilities of ChatGPT. Large language models have been of interest to companies in a variety of applications, including in developing chatbots. Another more rigorous need for large language models is in the notion of “enhanced troubleshooting,” by which we mean the ability of an individual at a company to query a centralized machine to find out why a certain failure scenario is occurring with, for example, equipment or materials. The concept behind enhanced troubleshooting is that a good deal of information is lost in industry as different individuals across different sites are faced with the same failure scenarios. Looking up why a particular problem may be occurring may require searching through a number of company databases that may not communicate with one another, or calling the right person at the company. It is also possible that an individual may not know who to contact, and consequently may end up solving a problem that has already been solved by another individual at the company all over again. These types of scenarios waste company resources and time.

LLM’s offer a potential direction for alleviating these issues. Specifically, they can have access to a centralized databased of company information from which they draw to provide answers to user queries in an automated workflow. This workflow may include accessing the database of information to pull out relevant documents, checking for relevancy, and then using the LLM to synthesize a response to the user’s query from the relevant documents. However, currently, this workflow would suffer from key challenges with LLM’s today, in particular that there is not a clear procedure, when engineering decision-making is in view, for assessing the relevancy of documents to an engineering task. Therefore, in this work, we discuss the enhanced troubleshooting workflow and the unique characteristics that it brings, such as the need to avoid fine-tuning, which limit the options for relevancy testing. We then discuss a number of different options for attempting to improve relevancy testing that involve claim simplification and evidence retrieval, pointwise [1] and listwise rerankers [2,3], sparse, dense, and hybrid searching [4] for the most relevant documents, and question-and-answer protocols [5] meant to bring out similarities and differences between information retrieved from the database and the original user’s question to aid with relevancy testing. We compare the results of using these different strategies for a variety of queries to evaluate the significance of these various potential components of the enhanced troubleshooting workflow to the overall goals of the method. Through this, we develop insights into how to create enhanced troubleshooting systems for engineering problems.

References:

[1] Ma, Xueguang, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. "Fine-tuning llama for multi-stage text retrieval." arXiv preprint arXiv:2310.08319 (2023).

[2] Ma, Xueguang, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. "Zero-shot listwise document reranking with a large language model." arXiv preprint arXiv:2305.02156 (2023).

[3] Pradeep, Ronak, Sahel Sharifymoghaddam, and Jimmy Lin. "RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!." arXiv preprint arXiv:2312.02724 (2023).

[4] Bhagdev, Ravish, Sam Chapman, Fabio Ciravegna, Vitaveska Lanfranchi, and Daniela Petrelli. "Hybrid search: Effectively combining keywords and semantic searches." In The Semantic Web: Research and Applications: 5th European Semantic Web Conference, ESWC 2008, Tenerife, Canary Islands, Spain, June 1-5, 2008 Proceedings 5, pp. 554-568. Springer Berlin Heidelberg, 2008.

[5] Yang, Jing, Didier Vega-Oliveros, Taís Seibt, and Anderson Rocha. "Explainable fact-checking through question answering." In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8952-8956. IEEE, 2022.