Scholarlink

<p>Large Language Models (LLMs) demonstrate remarkable capabilities in scientific tasks such as literature analysis and experimental design. For instance, these models excel at accurately extracting key findings from papers or generating coherent experimental procedures. However, existing evaluation benchmarks primarily assess performance using rich contextual inputs. We introduce LiveIdeaBench, a comprehensive benchmark evaluating LLMs’ scientific idea generation by assessing divergent thinking capabilities using single-keyword prompts. Drawing from Guilford’s creativity theory, our benchmark employs a dynamic panel of state-of-the-art LLMs to assess generated ideas across five dimensions: originality, feasibility, fluency, flexibility, and clarity. Through experimentation with over 40 leading models across 1180 keywords spanning 22 scientific domains, we reveal that the scientific idea generation capabilities measured by our benchmark are poorly predicted by standard metrics of general intelligence scores. Our results demonstrate that models like QwQ-32B-preview achieve creative performance comparable to models such as claude-3.7-sonnet: thinking, despite significant gaps in their general intelligence scores. These findings highlight the need for specialized evaluation benchmarks for scientific idea generation and suggest that enhancing these idea generation capabilities in LLMs may require different training strategies than those used for improving general problem-solving abilities. Such strategies could potentially enable a wider range of AI tools tailored for different stages of the scientific process.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating LLMs' divergent thinking capabilities for scientific idea generation with minimal context

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年3月7日

查看全文
  • Kai Ruan,
  • Xuan Wang,
  • Jixiang Hong,
  • Peng Wang,
  • Yang Liu,
  • Hao Sun

摘要

Large Language Models (LLMs) demonstrate remarkable capabilities in scientific tasks such as literature analysis and experimental design. For instance, these models excel at accurately extracting key findings from papers or generating coherent experimental procedures. However, existing evaluation benchmarks primarily assess performance using rich contextual inputs. We introduce LiveIdeaBench, a comprehensive benchmark evaluating LLMs’ scientific idea generation by assessing divergent thinking capabilities using single-keyword prompts. Drawing from Guilford’s creativity theory, our benchmark employs a dynamic panel of state-of-the-art LLMs to assess generated ideas across five dimensions: originality, feasibility, fluency, flexibility, and clarity. Through experimentation with over 40 leading models across 1180 keywords spanning 22 scientific domains, we reveal that the scientific idea generation capabilities measured by our benchmark are poorly predicted by standard metrics of general intelligence scores. Our results demonstrate that models like QwQ-32B-preview achieve creative performance comparable to models such as claude-3.7-sonnet: thinking, despite significant gaps in their general intelligence scores. These findings highlight the need for specialized evaluation benchmarks for scientific idea generation and suggest that enhancing these idea generation capabilities in LLMs may require different training strategies than those used for improving general problem-solving abilities. Such strategies could potentially enable a wider range of AI tools tailored for different stages of the scientific process.