In practical applications of crowd counting, developing efficient and lightweight models is crucial for addressing the challenges of resource-limited environments. However, many current approaches use large and complex architectures, resulting in significant computational and storage demands, which restrict their practical use in real-world situations. To address this challenge, this paper proposes an efficient and lightweight crowd counting network called SwiftCrowd. Specifically, we design a multi-scale architecture to handle crowd counting tasks, with SwiftFormer chosen as the backbone of the multi-scale network to minimize network parameters and computational costs. Building on this, a lightweight dual attention mechanism is utilized for additional feature processing to improve the network's feature extraction ability. Finally, SwiftCrowd integrates features from multiple scales through simple feature fusion operations. Through experiments on multiple crowd counting datasets, SwiftCrowd demonstrates outstanding counting accuracy and lightweight characteristics. Compared to existing methods, it exhibits significant competitiveness, showcasing its enormous potential for practical applications.