Information about an image’s source camera model is crucial in many forensic investigations, including protecting intellectual property and identifying internet users. Challenges arise when dealing with unknown camera models, which often exhibit only subtle feature differences-a task that extends beyond the reach of conventional CMI techniques. While state-of-the-art methods have achieved notable advancements, they tend to struggle with effectively managing difficult samples that contain considerable outliers, leading to significant performance degradation in complex scenarios. To address this issue, we propose a novel optimization approach, the Dual-Margin Triplet Loss (DMTL), to enhance the process of determining whether two images are from the same camera model in open-set scenarios. Our method extends the original triplet loss by integrating two additional margin constraints: a positive intra-class margin for intra-class compactness and a negative intra-class margin for inter-class separability. The dynamic collaboration of these margins enables DMTL to create clearer category boundaries and more discriminative embedding representations. Extensive experiments conducted on the Dresden and VISION datasets show that the DMTL method outperforms existing methodologies in terms of authentication accuracy for both known and unknown camera models.