Scholarlink

<p>The increasing demand for real-time, privacy-preserving video understanding on embedded and edge devices has exposed the limitations of existing video captioning models, which typically rely on large architectures and cloud-based inference. In this paper, we present GeoCaption, a lightweight, on-device multimodal video captioning framework that integrates visual, ambient audio, and GPS-based spatial signals to generate environmentally grounded captions under controlled GPS simulation settings. GeoCaption employs modality-specific lightweight encoders, including a MobileViT vision encoder, an OpenL3-CNN audio encoder, and a BiGRU-based GPS encoder with positional embeddings, whose representations are fused via a compact cross-modal transformer. Caption generation is performed using a MiniLM decoder trained through multimodal teacher-student distillation from a large CLIP-LLaVA mentor, enabling the transfer of both semantic and attention-level knowledge. To support deployment on microcontroller-class hardware, the model is optimized using quantization-aware training (INT8), achieving inference latency below 200 ms and memory usage under 40&#xa0;MB. Experiments on MSR-VTT, MSVD, and TGIF demonstrate that GeoCaption consistently outperforms lightweight baselines in CIDEr and BLEU while maintaining real-time performance on ARM Cortex-M and Cortex-A devices. These results demonstrate the feasibility of GeoCaption as a scalable multimodal captioning framework for TinyML and edge-AI applications.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

GeoCaption: a real-time, TinyML-optimized multimodal transformer for environmental video captioning using vision, audio, and GPS fusion

  • 期刊论文

  • 发表日期：

    2026年6月29日

查看全文
  • Qaisar Abbas,
  • Mubarak Albathan

摘要

The increasing demand for real-time, privacy-preserving video understanding on embedded and edge devices has exposed the limitations of existing video captioning models, which typically rely on large architectures and cloud-based inference. In this paper, we present GeoCaption, a lightweight, on-device multimodal video captioning framework that integrates visual, ambient audio, and GPS-based spatial signals to generate environmentally grounded captions under controlled GPS simulation settings. GeoCaption employs modality-specific lightweight encoders, including a MobileViT vision encoder, an OpenL3-CNN audio encoder, and a BiGRU-based GPS encoder with positional embeddings, whose representations are fused via a compact cross-modal transformer. Caption generation is performed using a MiniLM decoder trained through multimodal teacher-student distillation from a large CLIP-LLaVA mentor, enabling the transfer of both semantic and attention-level knowledge. To support deployment on microcontroller-class hardware, the model is optimized using quantization-aware training (INT8), achieving inference latency below 200 ms and memory usage under 40 MB. Experiments on MSR-VTT, MSVD, and TGIF demonstrate that GeoCaption consistently outperforms lightweight baselines in CIDEr and BLEU while maintaining real-time performance on ARM Cortex-M and Cortex-A devices. These results demonstrate the feasibility of GeoCaption as a scalable multimodal captioning framework for TinyML and edge-AI applications.