Scholarlink

<p>Decision support for inflammatory aneurysms demands guideline-concordant specificity. We compared three large-language-model systems—DeepSeek-R1, ChatGPT-4o and DeepSeek-V3—on accuracy and time efficiency for core diagnostic, interventional and surveillance decisions. In a prospective, rater-blinded evaluation, each model completed a 50-item instrument aligned to contemporary aortic guidance in five independent clean-context sessions (15 answer sets). Five consultant vascular specialists scored all outputs using a prespecified 0/1/2 rubric. The primary outcome was total accuracy (0–100); secondary outcomes were completion time and domain subscores. Inter-rater reliability was summarised with Fleiss’ κ and ICC(2,k). Mean total scores were 89.8 ± 1.8 (DeepSeek-R1), 88.4 ± 1.8 (ChatGPT-4o) and 77.8 ± 1.5 (DeepSeek-V3). One-way ANOVA showed a significant model effect (F(2,12) = 75.34, <i>p</i> = 1.61 × 10⁻⁷); Tukey contrasts indicated near-parity between R1 and 4o (Δ 1.44, <i>p</i> = 0.397) with both exceeding V3 (R1–V3 Δ 12.00, <i>p</i> = 2.80 × 10⁻⁷; 4o–V3 Δ 10.56, <i>p</i> = 1.13 × 10⁻⁶). Completion time separated strongly (R1 58.5 ± 0.95&#xa0;s; 4o 33.4 ± 0.46&#xa0;s; V3 19.8 ± 0.89&#xa0;s; F(2,12) = 3053, <i>p</i> = 5.69 × 10⁻¹⁷), and accuracy rose with time (Pearson <i>r</i> = 0.802, 95% CI 0.49–0.93, <i>p</i> = 3.24 × 10⁻⁴; Theil–Sen 0.285 points·s⁻¹). Differences concentrated in guideline-derived items (R1 36.44 ± 1.49; 4o 36.32 ± 1.45; V3 29.16 ± 0.86). Inter-rater agreement was substantial at item level (κ = 0.662) and excellent for totals (ICC[2,k] = 0.973); intra-model variability was ≈ 2%. In rule-dense, safety-critical decisions for inflammatory aneurysms, slower, reasoning-oriented decoding yields the highest accuracy, while a low-latency multimodal system achieves near-parity at markedly shorter times; a throughput-optimised model trades speed for omission-prone answers. Model selection should align with task criticality, with human verification retained.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Benchmarking LLM decision support in inflammatory aneurysms: DeepSeek-R1, DeepSeek-V3 and ChatGPT-4o

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年4月3日

查看全文
  • Jianqiang Hao,
  • Changwei Zhang

摘要

Decision support for inflammatory aneurysms demands guideline-concordant specificity. We compared three large-language-model systems—DeepSeek-R1, ChatGPT-4o and DeepSeek-V3—on accuracy and time efficiency for core diagnostic, interventional and surveillance decisions. In a prospective, rater-blinded evaluation, each model completed a 50-item instrument aligned to contemporary aortic guidance in five independent clean-context sessions (15 answer sets). Five consultant vascular specialists scored all outputs using a prespecified 0/1/2 rubric. The primary outcome was total accuracy (0–100); secondary outcomes were completion time and domain subscores. Inter-rater reliability was summarised with Fleiss’ κ and ICC(2,k). Mean total scores were 89.8 ± 1.8 (DeepSeek-R1), 88.4 ± 1.8 (ChatGPT-4o) and 77.8 ± 1.5 (DeepSeek-V3). One-way ANOVA showed a significant model effect (F(2,12) = 75.34, p = 1.61 × 10⁻⁷); Tukey contrasts indicated near-parity between R1 and 4o (Δ 1.44, p = 0.397) with both exceeding V3 (R1–V3 Δ 12.00, p = 2.80 × 10⁻⁷; 4o–V3 Δ 10.56, p = 1.13 × 10⁻⁶). Completion time separated strongly (R1 58.5 ± 0.95 s; 4o 33.4 ± 0.46 s; V3 19.8 ± 0.89 s; F(2,12) = 3053, p = 5.69 × 10⁻¹⁷), and accuracy rose with time (Pearson r = 0.802, 95% CI 0.49–0.93, p = 3.24 × 10⁻⁴; Theil–Sen 0.285 points·s⁻¹). Differences concentrated in guideline-derived items (R1 36.44 ± 1.49; 4o 36.32 ± 1.45; V3 29.16 ± 0.86). Inter-rater agreement was substantial at item level (κ = 0.662) and excellent for totals (ICC[2,k] = 0.973); intra-model variability was ≈ 2%. In rule-dense, safety-critical decisions for inflammatory aneurysms, slower, reasoning-oriented decoding yields the highest accuracy, while a low-latency multimodal system achieves near-parity at markedly shorter times; a throughput-optimised model trades speed for omission-prone answers. Model selection should align with task criticality, with human verification retained.