Scholarlink

Study Objectives: <p>We compared the performance of a comprehensive automated polysomnogram analysis algorithm—CAISR (Complete Artificial Intelligence Sleep Report)—to a multiexpert gold-standard panel, crowdsourced scorers, and experienced technicians for sleep staging and detecting arousals, respiratory events, and limb movements.</p> Methods: <p>A benchmark dataset of 57 polysomnogram records (Inter-Scorer Reliability dataset) with 200 30-second epochs scored per American Academy of Sleep Medicine guidelines was used. Annotations were obtained from (1) the American Academy of Sleep Medicine multiexpert gold-standard panel, (2) American Academy of Sleep Medicine Inter-Scorer Reliability (ISR) platform users (“crowd,” averaging 6,818 raters per epoch), (3) 3 experienced technicians, and (4) CAISR. Agreement was assessed via Cohen’s kappa (κ) and percent agreement.</p> Results: <p>Across tasks, CAISR achieved performance comparable to that of experienced technicians but did not match consensus-level agreement between the multiexpert gold standard and the crowd. For sleep staging, CAISR’s agreement with multiexpert gold standard was 82.1% (κ = 0.70), comparable to experienced technicians but below the crowd (κ = 0.88). Arousal detection showed 87.81% agreement (κ = 0.45), respiratory event detection 83.18% agreement (κ = 0.34), and limb movement detection 94.89% agreement (κ = 0.11), each aligning with performance equivalent to experienced technicians but trailing crowd agreement (κ = 0.83, 0.78, and 0.86 for detection of arousal, respiratory events, and limb movements, respectively).</p> Conclusions: <p>CAISR achieves experienced technician-level accuracy for polysomnogram scoring tasks but does not surpass the consensus-level agreement of a multiexpert gold standard or the crowd. These findings highlight the potential of automated scoring to match experienced technician-level performance while emphasizing the value of multirater consensus.</p> Citation: <p>Tripathi A, Nasiri S, Ganglberger W, et&#xa0;al. Automated analysis of the American Academy of Sleep Medicine Inter-Scorer Reliability gold-standard polysomnogram dataset. <i>J Clin Sleep Med.</i> 2025;21(11):1821–1829.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Automated analysis of the American Academy of Sleep Medicine Inter-Scorer Reliability gold-standard polysomnogram dataset

  • 期刊论文

  • 发表日期：

    2025年11月1日

查看全文
  • Ayush Tripathi,
  • Samaneh Nasiri,
  • Wolfgang Ganglberger,
  • Thijs Nassi,
  • Erik-Jan Meulenbrugge,
  • Haoqi Sun,
  • Katie L. Stone,
  • Emmanuel Mignot,
  • Dennis Hwang,
  • Lynn Marie Trotti,
  • Matthew A. Reyna,
  • Gari D. Clifford,
  • Umakanth Katwa,
  • Robert J. Thomas,
  • M. Brandon Westover

摘要

Study Objectives:

We compared the performance of a comprehensive automated polysomnogram analysis algorithm—CAISR (Complete Artificial Intelligence Sleep Report)—to a multiexpert gold-standard panel, crowdsourced scorers, and experienced technicians for sleep staging and detecting arousals, respiratory events, and limb movements.

Methods:

A benchmark dataset of 57 polysomnogram records (Inter-Scorer Reliability dataset) with 200 30-second epochs scored per American Academy of Sleep Medicine guidelines was used. Annotations were obtained from (1) the American Academy of Sleep Medicine multiexpert gold-standard panel, (2) American Academy of Sleep Medicine Inter-Scorer Reliability (ISR) platform users (“crowd,” averaging 6,818 raters per epoch), (3) 3 experienced technicians, and (4) CAISR. Agreement was assessed via Cohen’s kappa (κ) and percent agreement.

Results:

Across tasks, CAISR achieved performance comparable to that of experienced technicians but did not match consensus-level agreement between the multiexpert gold standard and the crowd. For sleep staging, CAISR’s agreement with multiexpert gold standard was 82.1% (κ = 0.70), comparable to experienced technicians but below the crowd (κ = 0.88). Arousal detection showed 87.81% agreement (κ = 0.45), respiratory event detection 83.18% agreement (κ = 0.34), and limb movement detection 94.89% agreement (κ = 0.11), each aligning with performance equivalent to experienced technicians but trailing crowd agreement (κ = 0.83, 0.78, and 0.86 for detection of arousal, respiratory events, and limb movements, respectively).

Conclusions:

CAISR achieves experienced technician-level accuracy for polysomnogram scoring tasks but does not surpass the consensus-level agreement of a multiexpert gold standard or the crowd. These findings highlight the potential of automated scoring to match experienced technician-level performance while emphasizing the value of multirater consensus.

Citation:

Tripathi A, Nasiri S, Ganglberger W, et al. Automated analysis of the American Academy of Sleep Medicine Inter-Scorer Reliability gold-standard polysomnogram dataset. J Clin Sleep Med. 2025;21(11):1821–1829.