Amidst globalization and digital transformation, foreign language acquisition has become a fundamental cognitive skill. Traditional vocabulary-focused methods, lacking contextual grounding and sensory integration, fail to develop communicative competence. This paper introduces an intelligent, multimodal learning system that integrates visual inputs with deep learning models. By combining Vision Transformer (ViT) for visual feature extraction and GPT-2 for contextual text generation, the system produces multilingual, semantically rich descriptions. It also employs text-to-speech for pronunciation support and includes interactive modules to reinforce retention. Operable offline, the system ensures data privacy and accessibility, promoting personalized, context-aware language learning through multimodal AI. This work contributes to a paradigm shift in computer-assisted language learning (CALL), emphasizing semantic grounding, learner autonomy, and cognitive engagement. Experimental results suggest promising usability in resource-constrained educational settings.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Multimodal AI Framework for Context-Aware Language Learning: Integrating Vision Transformer and GPT-2

  • Quang-Quy Tran,
  • Xuan-Truong Quach,
  • Ke-Luong Nguyen,
  • Thi-Tuyen Ho,
  • Van-Tho Ha,
  • The-Phat Nguyen

摘要

Amidst globalization and digital transformation, foreign language acquisition has become a fundamental cognitive skill. Traditional vocabulary-focused methods, lacking contextual grounding and sensory integration, fail to develop communicative competence. This paper introduces an intelligent, multimodal learning system that integrates visual inputs with deep learning models. By combining Vision Transformer (ViT) for visual feature extraction and GPT-2 for contextual text generation, the system produces multilingual, semantically rich descriptions. It also employs text-to-speech for pronunciation support and includes interactive modules to reinforce retention. Operable offline, the system ensures data privacy and accessibility, promoting personalized, context-aware language learning through multimodal AI. This work contributes to a paradigm shift in computer-assisted language learning (CALL), emphasizing semantic grounding, learner autonomy, and cognitive engagement. Experimental results suggest promising usability in resource-constrained educational settings.