Scholarlink

Vulnerability detectors based on language models (LMs) show promise, but their robustness under adversarial conditions remains underexplored. Specifically, prior evaluations often lack diverse adversarial attack strategies and sophisticated perturbation implementations. To address this, we present AdvCodeGen, a black-box adversarial framework that leverages a large language model (LLM) to systematically evaluate the robustness of various LM-based vulnerability detectors. Unlike purely Abstract syntax tree (AST)-based rule transformations, AdvCodeGen utilizes an LLM to not only implement 19 effective code perturbation rules but also to generate more diverse and syntactically varied adversarial examples while preserving functionality. We evaluated AdvCodeGen against six state-of-the-art LMs (CodeT5, CodeBERT, GraphCodeBERT, UnixCoder, StarCoder2, and CodeGen2.5) on the BigVul and Devign datasets. AdvCodeGen achieves attack success rates (ASR) up to 81.5% (BigVul/StarCoder2) and 60.3% (Devign/CodeGen2.5), significantly outperforming existing token-level adversarial methods (ALERT, CARROT, CODA, and AACEGEN) that typically achieve less than 40% ASR. Compared to naive LLM baselines, AdvCodeGen delivers 15–30% higher ASR while maintaining 97.19% compile success and 91.88% unit test success rates. These results demonstrate that rule-guided LLM code generation enables broader structural and lexical diversity, leading to more targeted and effective adversarial transformations. Our work provides a robust approach to evaluating and ultimately enhancing the security of LM-based vulnerability detectors.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

AdvCodeGen: Adversarial Code Generation via Large Language Models

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Eun Jung,
  • Jiacheng Li,
  • Yonghwi Kwon,
  • Hyoungshick Kim

摘要

Vulnerability detectors based on language models (LMs) show promise, but their robustness under adversarial conditions remains underexplored. Specifically, prior evaluations often lack diverse adversarial attack strategies and sophisticated perturbation implementations. To address this, we present AdvCodeGen, a black-box adversarial framework that leverages a large language model (LLM) to systematically evaluate the robustness of various LM-based vulnerability detectors. Unlike purely Abstract syntax tree (AST)-based rule transformations, AdvCodeGen utilizes an LLM to not only implement 19 effective code perturbation rules but also to generate more diverse and syntactically varied adversarial examples while preserving functionality. We evaluated AdvCodeGen against six state-of-the-art LMs (CodeT5, CodeBERT, GraphCodeBERT, UnixCoder, StarCoder2, and CodeGen2.5) on the BigVul and Devign datasets. AdvCodeGen achieves attack success rates (ASR) up to 81.5% (BigVul/StarCoder2) and 60.3% (Devign/CodeGen2.5), significantly outperforming existing token-level adversarial methods (ALERT, CARROT, CODA, and AACEGEN) that typically achieve less than 40% ASR. Compared to naive LLM baselines, AdvCodeGen delivers 15–30% higher ASR while maintaining 97.19% compile success and 91.88% unit test success rates. These results demonstrate that rule-guided LLM code generation enables broader structural and lexical diversity, leading to more targeted and effective adversarial transformations. Our work provides a robust approach to evaluating and ultimately enhancing the security of LM-based vulnerability detectors.