Multimodal large models, with strong understanding abilities, show broad industrial applications ability across domains. However, industrial scenarios impose stringent requirements for rapid model iteration under conditions of low data, low human resources, and low time costs, and existing training methods still fail to meet industrial demands in low-resource scenarios. To address these challenges, our paper constructs a high-quality multimodal industrial dataset and proposes a prompt tuning method based on sparse transformer. This method enables fast and efficient cross-modal prompt collaboration tailored to the industrial scenarios. Experimental results show that our method achieves excellent performance under low-resource conditions, verifying its effectiveness in resource-constrained scenarios. The main contributions include, firstly, the creation of the first Chinese industrial parts multimodal dataset generated by a hybrid multimodal data generation method. Secondly, the proposal of an efficient prompt tuning approach, which greatly improves the model’s performance under industrial low-resource conditions. Thirdly, the validation demonstrates that our method achieves outstanding results across various experiments. Our dataset is available at https://dx.doi.org/10.21227/3ngv-br47

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

STDC: Sparse Transformer Deep Collaboration Prompt Tuning for Industrial Multimodal Large Models

  • Yijun Bei,
  • Ke Wang,
  • Bin Zhao

摘要

Multimodal large models, with strong understanding abilities, show broad industrial applications ability across domains. However, industrial scenarios impose stringent requirements for rapid model iteration under conditions of low data, low human resources, and low time costs, and existing training methods still fail to meet industrial demands in low-resource scenarios. To address these challenges, our paper constructs a high-quality multimodal industrial dataset and proposes a prompt tuning method based on sparse transformer. This method enables fast and efficient cross-modal prompt collaboration tailored to the industrial scenarios. Experimental results show that our method achieves excellent performance under low-resource conditions, verifying its effectiveness in resource-constrained scenarios. The main contributions include, firstly, the creation of the first Chinese industrial parts multimodal dataset generated by a hybrid multimodal data generation method. Secondly, the proposal of an efficient prompt tuning approach, which greatly improves the model’s performance under industrial low-resource conditions. Thirdly, the validation demonstrates that our method achieves outstanding results across various experiments. Our dataset is available at https://dx.doi.org/10.21227/3ngv-br47