In the signature verification field, most existing methods rely on training with genuine and forgery signature pairs to improve accuracy. However, obtaining a large number of signatures, especially skilled forgery, from the same writer is unrealistic. In this paper, a multi-scale lightweight neural network structure suitable for few-shot handwritten signature recognition is firstly proposed, which integrates Ghost lightweight convolution and Efficient Channel Attention, which can comprehensively obtain multi-scale features of signatures while maintaining the model’s lightweight, thereby improving recognition accuracy. Secondly, a writer independent signature verification method is proposed, which does not require forgery signatures for training. By pre-training and one-shot episode training, the model learns more generalized features, achieving high accuracy in signature verification without relying on forged signatures. Finally, map the genuine and forgery signatures to the metric space, and verify the signature through distance metric. The experimental results on three publicly available datasets in different languages show that the proposed method can achieve competitive validation results, especially in cross dataset scenarios. For these three datasets, the accuracy of writer independent cross dataset validation has reached over 75%. Especially for the CEDAR dataset, cross dataset results can even reach over 99.50%.