Scholarlink

<p>We present a systematic evaluation of modern multimodal large language models (LLMs) for the classification of mean-motion and secular resonances from images of resonant arguments. Four benchmark datasets (RB-TEST, RB-PILOT, RB-SMALL, RB-FULL) were constructed to cover clear, ambiguous, and transient cases, with both binary and three-class outputs. Using standardized prompts (a full prompt for large models and a simplified variant for small models that cannot process complex instructions), we tested flagship commercial models, large open-source models, and small locally runnable models. Commercial LLMs reach <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(F_1=100\%\)</EquationSource> </InlineEquation> on simple cases and up to <InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(94\%\)</EquationSource> </InlineEquation> on the three-class RB-SMALL dataset, while the best open-source models also reach <InlineEquation ID="IEq3"> <EquationSource Format="TEX">\(100\%\)</EquationSource> </InlineEquation> on unambiguous cases and <InlineEquation ID="IEq4"> <EquationSource Format="TEX">\(76\%\)</EquationSource> </InlineEquation> on the complex ones. On the full binary benchmark, open-source models approach commercial performance (<InlineEquation ID="IEq5"> <EquationSource Format="TEX">\(F_1\approx 90\)</EquationSource> </InlineEquation>–<InlineEquation ID="IEq6"> <EquationSource Format="TEX">\(96\%\)</EquationSource> </InlineEquation>). Most errors occur in transient and resonance-sticking regimes. The results show that LLMs can perform resonance classification at levels comparable to those of classical or machine-learning methods without training or fine-tuning, and that even small open-source models achieve practically useful accuracy. The released benchmarks establish a reproducible standard for evaluating LLMs on dynamical astronomy tasks.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating multimodal commercial and open-source large language models for dynamical astronomy: a benchmark study of resonant behavior classification

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年3月28日

查看全文
  • Evgeny Smirnov,
  • Valerio Carruba

摘要

We present a systematic evaluation of modern multimodal large language models (LLMs) for the classification of mean-motion and secular resonances from images of resonant arguments. Four benchmark datasets (RB-TEST, RB-PILOT, RB-SMALL, RB-FULL) were constructed to cover clear, ambiguous, and transient cases, with both binary and three-class outputs. Using standardized prompts (a full prompt for large models and a simplified variant for small models that cannot process complex instructions), we tested flagship commercial models, large open-source models, and small locally runnable models. Commercial LLMs reach \(F_1=100\%\) on simple cases and up to \(94\%\) on the three-class RB-SMALL dataset, while the best open-source models also reach \(100\%\) on unambiguous cases and \(76\%\) on the complex ones. On the full binary benchmark, open-source models approach commercial performance ( \(F_1\approx 90\) – \(96\%\) ). Most errors occur in transient and resonance-sticking regimes. The results show that LLMs can perform resonance classification at levels comparable to those of classical or machine-learning methods without training or fine-tuning, and that even small open-source models achieve practically useful accuracy. The released benchmarks establish a reproducible standard for evaluating LLMs on dynamical astronomy tasks.