Class-agnostic counting (CAC) refers to a kind of counting method that estimates the number of objects in an image without knowing the explicit object categories. It is suitable for scenarios where the object types are diverse, unpredictable, or category information cannot be determined in advance, and it has significant implications for real-life applications. The main process of most CAC tasks involves extracting image features, performing similarity matching, and counting the objects. However, many current feature extraction methods struggle to simultaneously focus on multi-dimensional feature information, which limits the expressiveness of the features. Moreover, traditional CAC methods use fixed inner product calculations for similarity, which cannot dynamically adjust the similarity metrics based on the specific characteristics of the objects, leading to insufficient generalization capabilities. To address these issues, we propose the Channel Reconstruction and Dynamic Similarity Matching Network (CRDSMNet) for class-agnostic counting. It consists of three modules: (1) Scale-Prior Convolution-Driven Channel Reconstruction Feature Extraction Module, which embeds samples’ scale information into deformable convolutions, enabling the network’s receptive field to automatically adjust according to the scale of the given samples. It also groups channels, allowing each subgroup of sub-features to independently learn and extract specific feature information. (2) Dynamic Similarity Matching Module, which can dynamically learn channel attention weights, allowing the model to dynamically adjust attention according to the characteristics of different samples. (3) Iterative Attention Fusion Counting Module. Extensive comparative and ablation experiments on mainstream datasets (FSC147, CARPK) show the effectiveness of CRDSMNet in the CAC task.