Integrating visual transformers with biomechanical constraints for action quality assessment in competitive sports
摘要
Assessing how well an athlete performs a specific movement—rather than simply identifying what movement was performed—remains one of the harder open problems at the intersection of computer vision and sports science. We present a framework that marries visual Transformer architectures with biomechanical domain knowledge, targeting physics-informed action quality assessment (AQA) in competitive sports. Our approach rests on three pillars: a dual-stream architecture pairing Video Swin Transformer backbones with graph-based skeleton encoders, a multi-scale spatiotemporal module designed to capture motion dynamics at several temporal granularities simultaneously, and biomechanical constraint losses encoding joint angle feasibility, movement smoothness, and bilateral symmetry. A cross-modal attention mechanism allows the two streams to inform each other bidirectionally, producing unified embeddings that carry both perceptual and kinematic information. We validate the method on the MTL-AQA and FineDiving benchmarks, where it reaches Spearman correlation coefficients of 0.9512 and 0.9247, respectively—surpassing previous results on both datasets. Ablation studies confirm that each architectural component contributes measurably to performance gains. Compared with purely data-driven approaches that treat motion as unconstrained pixel sequences, our framework embeds physiological priors into the learning process, yielding predictions that respect anatomical feasibility. The biomechanical constraints simultaneously improve prediction accuracy and enable physically interpretable feedback—such as joint angle deviation reports and bilateral symmetry diagnostics—that coaches can translate into targeted training recommendations.