Scholarlink

Aiming at the triple crises of data inflation, hardware efficiency and environmental sustainability caused by the 260% annual increase in the parameter size of visual language macromodels (VLMs), this paper proposes the Spatio-Temporal Synergistic Sparse Transformer (VST2.0), which achieves efficient learning of cross-modal representations through algorithm-hardware-ecology co-design. The core innovations include: 1) Voronoi spatio-temporal entropy-based dynamic pruning for geometric perception, which reduces 41% of floating-point operations (FLOPs) on ViT-Huge and maintains visual feature fidelity; 2) enhanced learning-driven alignment mechanism for sparse distribution of graphics, which improves the R@1 of COCO cross-modal retrieval by 1.8% (67.5% → 69.3%), and reduces carbon emission by 15.6kW/h for a single inference; 3) enhanced learning-driven sparse distribution of graphs and texts, which reduces the carbon emissions by 1.6kW/h. 15.6kW/h; 3) Density metrics-guided hardware-aware computational graph reconstruction, which improves A100 TensorCores utilisation to 89%, and achieves an inference speed of 28 tokens/s (with 67% latency reduction). Experiments show that VST2.0 has 5.7% higher mAP@50 than the baseline model in medical imaging, industrial inspection, and autonomous driving scenarios, with the training cost reduced to $6,300/month (AWS p4d instance) and the edge inference cost is only $0.003/trip (Snapdragon Gen2 test). Theoretically established the first spatio-temporal pruning generalisation bound (R@1 ≥ 0.675–0.02⋅exp (-λgeo/2)) and hardware algorithmic synergy law (UTensorCore ∝ 1-e-0.75α). Promoting the development of green AI standards and building a carbon neutral lab with NVIDIA, which reduces CO₂12 tonnes of emissions from a single model per year (equivalent to planting 600 trees). This research provides a new paradigm for the lightweighting of large multimodal models, and its dynamic sparse coding mechanism can be extended to NeRF, robot perception and other fields, helping to achieve a balance between high-performance and low-carbon computing.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Spatio-Temporal Synergistic Sparse Transformer: Algorithm-Hardware-Ecosystem Co-design for Efficient Multimodal Learning

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Hongming Zhou,
  • Lu Yao,
  • Fenghao Ye,
  • Ziyao Zhang,
  • Anping Zhou

摘要

Aiming at the triple crises of data inflation, hardware efficiency and environmental sustainability caused by the 260% annual increase in the parameter size of visual language macromodels (VLMs), this paper proposes the Spatio-Temporal Synergistic Sparse Transformer (VST2.0), which achieves efficient learning of cross-modal representations through algorithm-hardware-ecology co-design. The core innovations include: 1) Voronoi spatio-temporal entropy-based dynamic pruning for geometric perception, which reduces 41% of floating-point operations (FLOPs) on ViT-Huge and maintains visual feature fidelity; 2) enhanced learning-driven alignment mechanism for sparse distribution of graphics, which improves the R@1 of COCO cross-modal retrieval by 1.8% (67.5% → 69.3%), and reduces carbon emission by 15.6kW/h for a single inference; 3) enhanced learning-driven sparse distribution of graphs and texts, which reduces the carbon emissions by 1.6kW/h. 15.6kW/h; 3) Density metrics-guided hardware-aware computational graph reconstruction, which improves A100 TensorCores utilisation to 89%, and achieves an inference speed of 28 tokens/s (with 67% latency reduction). Experiments show that VST2.0 has 5.7% higher mAP@50 than the baseline model in medical imaging, industrial inspection, and autonomous driving scenarios, with the training cost reduced to $6,300/month (AWS p4d instance) and the edge inference cost is only $0.003/trip (Snapdragon Gen2 test). Theoretically established the first spatio-temporal pruning generalisation bound (R@1 ≥ 0.675–0.02⋅exp (-λgeo/2)) and hardware algorithmic synergy law (UTensorCore ∝ 1-e-0.75α). Promoting the development of green AI standards and building a carbon neutral lab with NVIDIA, which reduces CO₂12 tonnes of emissions from a single model per year (equivalent to planting 600 trees). This research provides a new paradigm for the lightweighting of large multimodal models, and its dynamic sparse coding mechanism can be extended to NeRF, robot perception and other fields, helping to achieve a balance between high-performance and low-carbon computing.