Scholarlink

Objective <p>Applying large language models (LLM) in specialized medical disciplines presents unique challenges requiring precision, reliability, and domain-specific relevance. We evaluated a specialized LLM-driven system against neurosurgeons in vagus nerve stimulation (VNS) for drug-resistant epilepsy knowledge assessment—a complex neuromodulation therapy requiring transdisciplinary expertise in neural anatomy, epileptic disorders, and device technology.</p> Materials and methods <p>Thirty-six European neurosurgeons who completed a 2-day VNS masterclass were assessed using a multiple-choice questionnaire comprising 14 items with 67 binary propositions. We deployed open-source models—LLaMa 2 70B and MXBAI embedding model—using Neura, an AI infrastructure enabling transparent grounding through advanced retrieval augmented generation. The knowledge base consisted of 125 VNS-related publications curated by multidisciplinary faculty. Scoring ranged from -1 to + 1 per question. Performance was analyzed using Wilcoxon signed-rank tests, confusion matrices, and metrics including accuracy, precision, recall, and specificity.</p> Results <p>The AI achieved a score of 0.75, exceeding the highest individual clinician score (0.68; median: 0.50), with statistical significance (<i>p</i> &lt; 0.001). AI performed better in questions involving anatomical and technical information, while clinicians excelled in scenarios requiring practical judgment. Confusion matrices revealed higher true correct and true incorrect rates for AI, demonstrating perfect precision and specificity scores with no hallucinations detected.</p> Conclusions <p>Specialized LLM performance in this VNS knowledge assessment, coupled with its verifiability, points to promising applications across neurosurgical subspecialties for clinical decision support and education. The complementary strengths observed suggest that valuable implementations will emerge from synergistic approaches combining human experiential knowledge with AI's information processing capabilities across the broader field of neurosurgery.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Specialized AI and neurosurgeons in niche expertise: a proof-of-concept in neuromodulation with vagus nerve stimulation

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年7月25日

查看全文
  • Sami Barrit,
  • Giovanni Ranuzzi,
  • Steffen Fetzer,
  • Mejdeddine Al Barajraji,
  • Salim El Hadwe,
  • Marc Zanello,
  • Martin Ortler,
  • Julieta O’Flaherty,
  • Nicolas Massager,
  • Joseph R. Madsen,
  • Maxine Dibué,
  • Romain Carron

摘要

Objective

Applying large language models (LLM) in specialized medical disciplines presents unique challenges requiring precision, reliability, and domain-specific relevance. We evaluated a specialized LLM-driven system against neurosurgeons in vagus nerve stimulation (VNS) for drug-resistant epilepsy knowledge assessment—a complex neuromodulation therapy requiring transdisciplinary expertise in neural anatomy, epileptic disorders, and device technology.

Materials and methods

Thirty-six European neurosurgeons who completed a 2-day VNS masterclass were assessed using a multiple-choice questionnaire comprising 14 items with 67 binary propositions. We deployed open-source models—LLaMa 2 70B and MXBAI embedding model—using Neura, an AI infrastructure enabling transparent grounding through advanced retrieval augmented generation. The knowledge base consisted of 125 VNS-related publications curated by multidisciplinary faculty. Scoring ranged from -1 to + 1 per question. Performance was analyzed using Wilcoxon signed-rank tests, confusion matrices, and metrics including accuracy, precision, recall, and specificity.

Results

The AI achieved a score of 0.75, exceeding the highest individual clinician score (0.68; median: 0.50), with statistical significance (p < 0.001). AI performed better in questions involving anatomical and technical information, while clinicians excelled in scenarios requiring practical judgment. Confusion matrices revealed higher true correct and true incorrect rates for AI, demonstrating perfect precision and specificity scores with no hallucinations detected.

Conclusions

Specialized LLM performance in this VNS knowledge assessment, coupled with its verifiability, points to promising applications across neurosurgical subspecialties for clinical decision support and education. The complementary strengths observed suggest that valuable implementations will emerge from synergistic approaches combining human experiential knowledge with AI's information processing capabilities across the broader field of neurosurgery.