Entropy-Aware Fusion Speculative Decoding for Reliable and Efficient Domain Text Generation
摘要
Large language models (LLMs) have achieved impressive performance in general language generation and reasoning. However, applying large, general-purpose LLMs to domain-specific tasks is often impractical due to their limited domain knowledge, high inference cost, and deployment complexity. In contrast, domain-specialized smaller LLMs—adapted via parameter-efficient fine-tuning—offer better efficiency and domain alignment, but suffer from limited generalization and reduced generation quality. To address this trade-off, we propose Entropy-Aware Fusion Speculative Decoding (EnFuS), a collaborative decoding framework that couples a domain-specialized small LLM with a general-purpose large LLM. EnFuS integrates two key components: (1) a conditional entropy-based probability fusion mechanism that dynamically weights model predictions based on confidence, and (2) a two-stage speculative decoding process that accelerates generation via fast token acceptance and fallback verification. Experiments on four benchmarks (WikiText, HumanEval, AlpacaEval, GSM8k) show that EnFuS improves consistency and coherence by up to 6.5% and achieves up to 3 \(\times \) inference speedup over strong baselines. EnFuS is model-agnostic, plug-and-play, and generalizes well across model families, offering a practical solution for reliable and efficient domain text generation. We release our code and data at https://github.com/EasonEleven/EnFuS to facilitate further research.