Solving geometry problems remains a major challenge for multi-modal large language models (MLLMs), primarily due to their reliance on extensive, high-quality datasets and their limited capacity to generalize intermediate reasoning steps. To overcome these issues, we introduce GeoGRPO (Geometry-aware Group Relative Policy Optimization), a two-phase training approach designed to explicitly guide and optimize each stage of the reasoning process. In the first phase, we conduct supervised fine-tuning of a Qwen2.5-VL-7B policy model using our step-structured Geometry170K dataset, enhancing its capability to generate organized geometric proofs. In the second phase, we propose a novel Process Reward Model that assesses each generated reasoning step against expert solutions, providing fine-grained rewards. These detailed rewards are integrated into a newly designed GRPO objective, which steers the policy training towards step-level accuracy and logical consistency. When evaluated on datasets including GeoQA, Geometry3K, MathVista, and PGPS9K, GeoGRPO-Qw-7B achieves top-1 accuracies of 80.6%, 53.2%, 73.4%, and 50.6%, respectively, significantly outperforming baseline models. Furthermore, among open-source LLMs/MLLMs of similar scale, GeoGRPO-Qw-7B consistently leads in performance across all benchmarks, highlighting the effectiveness of fine-grained process supervision in enhancing geometric reasoning capabilities.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

GeoGRPO: Investigating the Stepwise-GRPO Enhancement in RLHF Framework

  • Kecheng Liang,
  • Xinyu Li,
  • Weixing Chen,
  • Yang Liu

摘要

Solving geometry problems remains a major challenge for multi-modal large language models (MLLMs), primarily due to their reliance on extensive, high-quality datasets and their limited capacity to generalize intermediate reasoning steps. To overcome these issues, we introduce GeoGRPO (Geometry-aware Group Relative Policy Optimization), a two-phase training approach designed to explicitly guide and optimize each stage of the reasoning process. In the first phase, we conduct supervised fine-tuning of a Qwen2.5-VL-7B policy model using our step-structured Geometry170K dataset, enhancing its capability to generate organized geometric proofs. In the second phase, we propose a novel Process Reward Model that assesses each generated reasoning step against expert solutions, providing fine-grained rewards. These detailed rewards are integrated into a newly designed GRPO objective, which steers the policy training towards step-level accuracy and logical consistency. When evaluated on datasets including GeoQA, Geometry3K, MathVista, and PGPS9K, GeoGRPO-Qw-7B achieves top-1 accuracies of 80.6%, 53.2%, 73.4%, and 50.6%, respectively, significantly outperforming baseline models. Furthermore, among open-source LLMs/MLLMs of similar scale, GeoGRPO-Qw-7B consistently leads in performance across all benchmarks, highlighting the effectiveness of fine-grained process supervision in enhancing geometric reasoning capabilities.