With the rapid development of communication technology and the increasing number of wireless devices, the issue of communication security in the Internet of Things (IoT) is becoming increasingly important. To accurately and efficiently implement device identity authentication, a Specific Emitter Identification (SEI) method from the physical layer perspective has been proposed. However, existing methods largely focus on closed-set recognition (OSR), where training is conducted on known categories and testing is done on the same categories, which cannot be applied in real-world scenarios. In the presence of unknown categories not seen during training, these methods are prone to misjudgment, affecting communication security. Therefore, this paper proposes an open-set recognition (OSR) framework called DGEGAN-OpenMax, which combines Dynamic Group Equivariant Convolutional Networks and Generative Adversarial Networks with OpenMax (GAN-OpenMax). This framework enhances the capability of embedding representations of communication signals in complex environments and utilizes GANs to simulate unknown data distributions for retraining the model, thus improving the discriminative ability against unknown domains. Experimental results show that, under conditions of 100 known classes and 20 unknown classes, the proposed method achieves an F1 score of 0.8 at −4 dB and above, outperforming other OSR methods.