<p>Most existing methods rely on the similarity correlation matrix to characterize the repetitiveness of actions, but their scalability is hindered due to the quadratic computational complexity. In this work, we introduce a novel approach that employs an action query representation to localize class-agnostic repeated action cycles with linear computational complexity. Based on this representation, we develop two key components to tackle the essential challenges of temporal repetition counting. Firstly, to tackle open-set action counting, we define two action classes: “repetitive actions” and “others”. Instead of manually defining the repetitive action class, we propose a dynamic action query strategy. Here, each action query directly represents an extracted video feature, allowing the repetitive actions of interest to be dynamically defined based on the video content itself. Secondly, to distinguish these repetitive action queries from others, we propose inter-query contrastive learning. This performs contrastive clustering over the queries, pulling similar action patterns together while pushing apart those related to background or unrelated movements. As a result, queries classified as “repetitive actions” are considered as repetitive cycles, which are then used for counting. Thanks to the query-based representation and contrastive learning strategy, our method significantly outperforms previous works on accuracy while being more lightweight and time-efficient. On the challenging RepCountA benchmark, we outperform the state-of-the-art method TransRAC by 26.5% in OBO accuracy, with a 22.7% mean error decrease and 94.1% computational burden reduction. Code and models are publicly available at project page.</p>