Evaluating the Usability of a Chatbot-Driven Operational Manual for Maintenance Professionals in Manufacturing Industry
摘要
This paper presents a usability test of a chatbot designed for the operational manual of a machining center within a manufacturing industry context as well as a survey of maintenance repairmen, focusing on information retrieval. The usability test was conducted with maintenance professionals experienced in working with similar machines and operational manuals. The test is divided into two parts. The first part evaluates how quickly participants can retrieve information using three different media—a printed version, a PDF file, and the developed chatbot. The second part covers a follow-up interview with the test participants. Twelve test participants with varying levels of experience in generative artificial intelligence (most with little or no experience) were involved in the study. Despite the limited familiarity with AI tools, the results favor the chatbot in terms of average information retrieval time. On average, participants took 40 s to find the requested information using the chatbot, compared to 1 min and 35 s with a searchable PDF file, and 2 min and 27 s with a printed manual. While participants generally expressed a positive attitude towards the use of the chatbot, they also raised concerns regarding its future applicability and potential limitations. These concerns will be discussed in detail in the paper. The survey shows that there is much efficiency to gain as the average maintenance repairman at the case company spend almost 2.5 h per week retrieving this type of information.