<p>Recent advancements in Large Language Models (LLMs) have significantly enhanced end-to-end voice-based human-computer interaction. However, automatic speech recognition systems remain a critical bottleneck due to their susceptibility to noise and limitations in handling contextual dependencies. Existing text correction methods based on LLMs often depend on the inherent performance of the base model, typically characterized by a larger parameter scale, which incur significant computational costs. To address these challenges, we propose Contextual Chain-of-Thought (CCoT), a novel framework inspired by cognitive reasoning processes. CCoT integrates contextual understanding and sequential reasoning to effectively address the limitations of current correction methods in handling contextual errors and elucidating the causes of errors in lengthy texts. As part of this work, we constructed PALTEC, a large-scale Chinese long-text error correction dataset containing 0.1 million samples from diverse domains, which was used for supervised fine-tuning of small-scale LLMs. By leveraging a teacher-student learning paradigm enhanced with knowledge distillation, CCoT facilitates large-scale LLMs in extracting context and cognitive insights from long texts. This process guides smaller-scale LLMs to develop advanced reasoning capabilities. Experimental results demonstrate that CCoT achieves advanced performance. Remarkably, CCoT achieves these results using a model with only 0.5 billion parameters, highlighting its efficiency and effectiveness in low cost addressing contextual and long-text error correction challenges.</p>