Scholarlink

Background <p>Symptomatic lumbar spinal stenosis (LSS) is a disabling condition with a substantial economic impact. The determination of surgical candidacy for LSS relies on a subjective assessment of multiple clinical and imaging factors, leading to variability in recommendations. Artificial intelligence (AI), including traditional machine learning (ML) and emerging large language models (LLMs), holds promise for enhancing the accuracy and consistency of surgical decision-making. However, the comparative performance of LLMs against established ML methods for this specific task remains unclear. This study therefore benchmarks various LLM configurations against a conventional random forest model in predicting LSS surgery recommendation using structured clinical vignettes.</p> Methods <p>Various LLMs, including zero-shot ChatGPT3.5, fine-tuned GPT-NEO (2.7B and 125&#xa0;M parameters), and BioMedLM, were benchmarked against a random forest model. Using a ground truth established by five spine surgeons from 500 synthetic medical vignettes (36 variables each), models provided surgical recommendations. LLMs received these vignette data in text format; the random forest received structured tabular format, to simulate realistic LSS surgical candidates. Performance was evaluated using root mean square error (RMSE) for regression and area under the receiver operating curve (AUROC) for binary classification.</p> Results <p>For surgical recommendation, zero-shot ChatGPT3.5 achieved an RMSE of 0.37 (AUROC = 0.79). The random forest model, trained on the structured data, achieved superior performance (RMSE = 0.12, AUROC = 0.96). Fine-tuned LLMs showed improved accuracy: GPT-NEO 125&#xa0;m achieved an RMSE of 0.224 (AUROC = 0.81), BioMedLM an RMSE of 0.258 (AUROC = 0.86), and GPT-NEO 2.7b achieved an RMSE of 0.275 (AUROC = 0.90).</p> Conclusions <p>When benchmarked on structured clinical vignettes, LLMs demonstrate credible but inferior predictive accuracy for LSS surgery recommendation compared to a conventional random forest model. These findings highlight that for well-structured data, specialized ML models retain an advantage. The critical next step is to evaluate LLMs on unstructured, real-world clinical data where their natural language processing strengths may be decisive.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Benchmarking of large language models to determine candidacy for spine surgery and comparison with conventional machine learning

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年4月9日

查看全文
  • Christopher Rice,
  • Frederik Abel,
  • Alemu Sisay Nigru,
  • Andrej Rusakov,
  • Nikolai S. Nikolaev,
  • Yurii Baiun,
  • Alessandra Falk,
  • Titenkov Yuriy,
  • Frolov Valerii,
  • Aleksey Kopeev,
  • Raphael Mourad

摘要

Background

Symptomatic lumbar spinal stenosis (LSS) is a disabling condition with a substantial economic impact. The determination of surgical candidacy for LSS relies on a subjective assessment of multiple clinical and imaging factors, leading to variability in recommendations. Artificial intelligence (AI), including traditional machine learning (ML) and emerging large language models (LLMs), holds promise for enhancing the accuracy and consistency of surgical decision-making. However, the comparative performance of LLMs against established ML methods for this specific task remains unclear. This study therefore benchmarks various LLM configurations against a conventional random forest model in predicting LSS surgery recommendation using structured clinical vignettes.

Methods

Various LLMs, including zero-shot ChatGPT3.5, fine-tuned GPT-NEO (2.7B and 125 M parameters), and BioMedLM, were benchmarked against a random forest model. Using a ground truth established by five spine surgeons from 500 synthetic medical vignettes (36 variables each), models provided surgical recommendations. LLMs received these vignette data in text format; the random forest received structured tabular format, to simulate realistic LSS surgical candidates. Performance was evaluated using root mean square error (RMSE) for regression and area under the receiver operating curve (AUROC) for binary classification.

Results

For surgical recommendation, zero-shot ChatGPT3.5 achieved an RMSE of 0.37 (AUROC = 0.79). The random forest model, trained on the structured data, achieved superior performance (RMSE = 0.12, AUROC = 0.96). Fine-tuned LLMs showed improved accuracy: GPT-NEO 125 m achieved an RMSE of 0.224 (AUROC = 0.81), BioMedLM an RMSE of 0.258 (AUROC = 0.86), and GPT-NEO 2.7b achieved an RMSE of 0.275 (AUROC = 0.90).

Conclusions

When benchmarked on structured clinical vignettes, LLMs demonstrate credible but inferior predictive accuracy for LSS surgery recommendation compared to a conventional random forest model. These findings highlight that for well-structured data, specialized ML models retain an advantage. The critical next step is to evaluate LLMs on unstructured, real-world clinical data where their natural language processing strengths may be decisive.