Over recent years, Mixture-of-Experts (MoE) models have emerged as a prominent research focus. In comparison to conventional dense large language models (LLMs), MoE architectures deliver superior performance while employing a reduced number of active parameters. Nevertheless, the efficient and precise allocation of experts to individual tokens within the MoE layer remains an enduring challenge in the field. To address this challenge, we propose LACD-MoE, which incorporates a layer-aware adaptive routing strategy alongside a confidence-difference constraint, improving model performance and enhancing the efficiency of computational resource utilization. In contrast to prior routing methods that activate a fixed number of experts, LACD-MoE dynamically adjusts the number of activated experts based on token-specific routing decisions. This adaptive strategy enables more efficient allocation of computational resources based on the complexity of individual tokens. Compared to existing dynamic routing methods, our approach explicitly models the relationship between expert activation and model depth, while incorporating a confidence-difference constraint to enable more precise and efficient expert selection. Extensive experiments across multiple datasets show that LACD-MoE outperforms state-of-the-art approaches by an average margin of 4.41%–8.50%, while also enhancing computational efficiency.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

LACD-MoE: Layer-Aware Adaptive Routing with Confidence-Difference Constraints for MoE Models

  • Liqi Jiang,
  • Cuifeng Gao,
  • Junmin Wu

摘要

Over recent years, Mixture-of-Experts (MoE) models have emerged as a prominent research focus. In comparison to conventional dense large language models (LLMs), MoE architectures deliver superior performance while employing a reduced number of active parameters. Nevertheless, the efficient and precise allocation of experts to individual tokens within the MoE layer remains an enduring challenge in the field. To address this challenge, we propose LACD-MoE, which incorporates a layer-aware adaptive routing strategy alongside a confidence-difference constraint, improving model performance and enhancing the efficiency of computational resource utilization. In contrast to prior routing methods that activate a fixed number of experts, LACD-MoE dynamically adjusts the number of activated experts based on token-specific routing decisions. This adaptive strategy enables more efficient allocation of computational resources based on the complexity of individual tokens. Compared to existing dynamic routing methods, our approach explicitly models the relationship between expert activation and model depth, while incorporating a confidence-difference constraint to enable more precise and efficient expert selection. Extensive experiments across multiple datasets show that LACD-MoE outperforms state-of-the-art approaches by an average margin of 4.41%–8.50%, while also enhancing computational efficiency.