Source camera device linking plays a crucial role in image forensics to verify whether two digital images were taken by the same camera device. While significant performance has been made by the PRNU-based correlation methods over the past decade, the primary challenge remains unresolved in scenarios with small-sized images. In this paper, we formulate the task of source camera device linking as a binary classification problem and propose a novel framework, hierarchical residual Siamese networks, to solve the challenge of small-sized images. By leveraging the encoder-decoder architecture with skip connections and the capabilities of hierarchical residual blocks, the proposed hierarchical residual network achieves hierarchical multi-scale feature aggregation across spatial and channel dimensions, facilitating the extraction of subtle sensor pattern noises from input images. Compared with the existing state-of-the-art methods, including both the PRNU-based correlation methods and deep learning-based methods, our proposed method not only achieves the best overall performance but also demonstrates a superior ability to balance sensitivity and specificity, thereby providing a more reliable and less biased classification.