MoIST: Mixture of Intellectuals Via Student-Teachers
摘要
In this work, we introduce MoIST (Mixture of Intellectuals via Student-Teachers), a novel framework that combines Knowledge Distillation (KD) and Mixture of Experts (MoE) to improve model efficiency. MoIST distills knowledge from a large teacher model into multiple smaller, specialized student models, which are then routed to handle specific subsets of the dataset. This routing mechanism, inspired by MoE, improves computational efficiency while maintaining model performance. We explore various architectural configurations for both the student models and the routing mechanism, demonstrating that MoIST can achieve accuracy comparable to the teacher model while significantly decreasing computational cost. Our results show that MoIST provides a promising approach to training an extra-efficient model, particularly in environments with limited resources.