Scholarlink

<p>Pre-trained large language models (LLMs) have been widely applied to natural language-based code generation. However, because code generation tasks are highly sensitive to structured information and exhibit diverse logical forms, directly adapting methods optimized for natural language tasks to code generation often proves less effective for open-source models. Conventional fine-tuning is generally limited to single, specific scenarios, while the standard Chain-of-Thought (CoT) approach differs from code in both expression and logical structure. Consequently, existing methods lack code-specific optimization strategies for downstream tasks. To address these issues, this paper leverages the unique logical interpretation and structured information of programming languages, specifically, pseudocode reflecting abstract syntax features, to construct a unified code generation framework called UCP, which integrates multi-task learning and enhanced alignment. The goal is to improve the code generation capabilities of open-source LLMs. By designing tasks that generate code from pseudocode, we build a multi-task dataset to jointly fine-tune LLMs and apply reinforcement learning(RL), optimizing a multi-task loss function to balance data distribution and task difficulty. We also establish consistent learning objectives at each stage through task-aligned training. Furthermore, focusing on code completion, we construct a multi-granularity code completion dataset—encompassing function-level, line-level, and token-level completions—for use in the RL phase. At the inference stage, we adopt a cyclic self-filling inference mechanism aligned with the completion task. We also design fill-in-the-blank tasks during pre-training to ensure alignment between the training and inference stages. The Qwen2.5-Coder-7B-Instruct model was post-trained using our proposed UCP, resulting in a HumanEval pass@1 metric improvement from 84.15 to 87.80% and a LiveCodeBench pass@1 metric increase from 25.4 to 27.2%, both surpassing other mainstream open-source models with similar parameter scales. By exposing intermediate pseudocode, UCP enhances interpretability and can benefit AI-powered IDEs (e.g., Cursor), improving developer trust and real-world usability.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

UCP: a unified framework for code generation with pseudocode-based multi-task learning and reinforcement alignment

  • 期刊论文

  • 发表日期：

    2025年6月10日

查看全文
  • Yongjun Wen,
  • Zhihao Cui,
  • Yihao Liu,
  • Zhao Zhang,
  • Jiake Zhou,
  • Lijun Tang

摘要

Pre-trained large language models (LLMs) have been widely applied to natural language-based code generation. However, because code generation tasks are highly sensitive to structured information and exhibit diverse logical forms, directly adapting methods optimized for natural language tasks to code generation often proves less effective for open-source models. Conventional fine-tuning is generally limited to single, specific scenarios, while the standard Chain-of-Thought (CoT) approach differs from code in both expression and logical structure. Consequently, existing methods lack code-specific optimization strategies for downstream tasks. To address these issues, this paper leverages the unique logical interpretation and structured information of programming languages, specifically, pseudocode reflecting abstract syntax features, to construct a unified code generation framework called UCP, which integrates multi-task learning and enhanced alignment. The goal is to improve the code generation capabilities of open-source LLMs. By designing tasks that generate code from pseudocode, we build a multi-task dataset to jointly fine-tune LLMs and apply reinforcement learning(RL), optimizing a multi-task loss function to balance data distribution and task difficulty. We also establish consistent learning objectives at each stage through task-aligned training. Furthermore, focusing on code completion, we construct a multi-granularity code completion dataset—encompassing function-level, line-level, and token-level completions—for use in the RL phase. At the inference stage, we adopt a cyclic self-filling inference mechanism aligned with the completion task. We also design fill-in-the-blank tasks during pre-training to ensure alignment between the training and inference stages. The Qwen2.5-Coder-7B-Instruct model was post-trained using our proposed UCP, resulting in a HumanEval pass@1 metric improvement from 84.15 to 87.80% and a LiveCodeBench pass@1 metric increase from 25.4 to 27.2%, both surpassing other mainstream open-source models with similar parameter scales. By exposing intermediate pseudocode, UCP enhances interpretability and can benefit AI-powered IDEs (e.g., Cursor), improving developer trust and real-world usability.