Recent developments in Large Language Models bring a personal AI coach closer to reality. However, to apply this to sport activities, multimodal LLMs still struggle to provide precise technical advice from videos. This study demonstrates that Graph Convolutional Networks (GCN) can help bridge the gap between human and AI coaching by extracting meaningful features from human skeletons. Using limited data consisting only of the player’s pose without ball or racket tracking, GCNs show strong capabilities to detect events frame-wise and correctly classify them, through learning new topologies and spatio-temporal relationships between physically distant joints. The proposed framework uses a two-stage approach: first, a Channel-wise Topology Refinement-GCN (CTR-GCN) is trained on professional athletes to learn canonical tennis motions and strikes; second, the backbone is frozen and transfer learning is applied to identify technical errors and their root-cause joints. To support this, we introduce a dataset of in-the-wild videos from amateur tennis players with semi-automated error annotations.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Shot and Technical Error Detection in Tennis Using Graph Convolutional Networks

  • Pierre Hellich,
  • Jiang Kan,
  • Jin Song Dong,
  • Mathieu Salzmann

摘要

Recent developments in Large Language Models bring a personal AI coach closer to reality. However, to apply this to sport activities, multimodal LLMs still struggle to provide precise technical advice from videos. This study demonstrates that Graph Convolutional Networks (GCN) can help bridge the gap between human and AI coaching by extracting meaningful features from human skeletons. Using limited data consisting only of the player’s pose without ball or racket tracking, GCNs show strong capabilities to detect events frame-wise and correctly classify them, through learning new topologies and spatio-temporal relationships between physically distant joints. The proposed framework uses a two-stage approach: first, a Channel-wise Topology Refinement-GCN (CTR-GCN) is trained on professional athletes to learn canonical tennis motions and strikes; second, the backbone is frozen and transfer learning is applied to identify technical errors and their root-cause joints. To support this, we introduce a dataset of in-the-wild videos from amateur tennis players with semi-automated error annotations.