While a well-trained Transformer-based Large Language model has been proven to own an excellent ability to answer any question based on the input, the high accuracy of the answer output, especially for testing procedure generation based on testing requirements for automotive components focused in this paper, usually relies on either pre-training data with domain knowledge about the testing or input prompts with clear requirement instruction by well-dedicated prompt engineering. Since expensive pre-training data or instructing an overwhelming volume of untraceable tacit expertise in the input prompts is required in those general LLMs for procedure generation, it is not easy for most industries to adopt LLMs for their practical applications. This paper explores the model performance for the case when only limited domain-specific training data, around two hundred records, is available for our chosen models with fine-tuning and further provides insight into what level we can rely on fine-tuning for testing procedure generation based on Testing requirements for automotive components with BLEU evaluation. We chose Flan-T5-large and Mistral-7B as our target models with consideration for the model sizes with affordable fine-tuning methods and their few-shot prediction performance. Our results show that both fine-tuned target models can generate our expected procedures with more than 0.8 BLEU on average, while Mistral-7B can even derive new content for the procedure.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Testing Procedure Generation Based on Testing Requirements for Automotive Components with LLM Fine-Tuning

  • Man Yiu Chow,
  • Mitsuhiro Kitani

摘要

While a well-trained Transformer-based Large Language model has been proven to own an excellent ability to answer any question based on the input, the high accuracy of the answer output, especially for testing procedure generation based on testing requirements for automotive components focused in this paper, usually relies on either pre-training data with domain knowledge about the testing or input prompts with clear requirement instruction by well-dedicated prompt engineering. Since expensive pre-training data or instructing an overwhelming volume of untraceable tacit expertise in the input prompts is required in those general LLMs for procedure generation, it is not easy for most industries to adopt LLMs for their practical applications. This paper explores the model performance for the case when only limited domain-specific training data, around two hundred records, is available for our chosen models with fine-tuning and further provides insight into what level we can rely on fine-tuning for testing procedure generation based on Testing requirements for automotive components with BLEU evaluation. We chose Flan-T5-large and Mistral-7B as our target models with consideration for the model sizes with affordable fine-tuning methods and their few-shot prediction performance. Our results show that both fine-tuned target models can generate our expected procedures with more than 0.8 BLEU on average, while Mistral-7B can even derive new content for the procedure.