<p>The rapid evolution of malware increasingly undermines traditional static analysis and signature-based detection methods, particularly in malware open-set recognition (MOSR), where the dual challenge of classifying known families and detecting unknown samples is paramount. To tackle this, we propose Ensemble Multi-Stream Threshold Network (EMSTNet), a novel multi-stream fusion network designed for MOSR, leveraging hybrid features and integrating three synergistic modules: metric learning, dynamic threshold optimization, and ensemble learning. Metric learning maps diverse malware features—including static, dynamic, and API call characteristics—into a unified embedding space, enhancing feature discriminability and inter-feature relationships. A dynamic threshold optimization mechanism optimizes classification boundaries for known families while robustly identifying unknown samples through adaptive confidence and boundary tuning. Additionally, an ensemble learning strategy employs soft voting to fuse predictions from multiple feature-specific classifiers, boosting accuracy and resilience against imbalanced and complex distributions. EMSTNet achieves state-of-the-art performance in experiments across diverse malware datasets, with a classification accuracy of 95.53% for known families and a detection accuracy of 92.72% for unknown samples, surpassing existing approaches, especially in challenging open-set and imbalanced scenarios. Furthermore, in a challenging cross-dataset evaluation where unknown samples are drawn from an independent external source, the framework maintains robust performance with 90.92% classification accuracy and 86.40% detection accuracy, demonstrating strong generalization capability. This framework advances MOSR by offering a scalable and adaptable solution, with future efforts aimed at enhancing real-time adaptability and self-learning capabilities for dynamic cybersecurity applications.</p>