Scholarlink

Introduction <p>HIV and sexually transmitted infections (STIs) continue to pose significant public health challenges globally. <i>MySTIRisk</i>, developed at Melbourne Sexual Health Centre (MSHC), is a machine learning-based tool that predicts individual risk for HIV, syphilis, gonorrhoea, and chlamydia using demographic and behavioural data. While initial validation showed promising results, external validation is crucial to assess its generalisability. This study externally validates <i>MySTIRisk</i> using data from the Sydney Sexual Health Centre (SSHC), Australia’s second-largest sexual health centre.</p> Methods <p>Following TRIPOD guidelines, we analysed consultations from patients aged 18 years and older attending SSHC between January 2013 and December 2023. Pre-trained <i>MySTIRisk</i> models were applied directly without modification. Performance was evaluated using the area under the receiver operating characteristic curve (AUC), sensitivity, and specificity at multiple thresholds, with subgroup analyses across demographic characteristics.</p> Results <p>We analysed 159,043 to 207,582 consultations at SSHC, depending on the specific infections tested. The median age was 30 years, and 60.2% to 68.8% of the consultations involved men who have sex with men. The area under the receiver operating characteristic curve (AUC) values using data from SSHC were 0.67 (95% CI: 0.65–0.68) for HIV, 0.70 (95% CI: 0.69–0.71) for syphilis, 0.73 (95% CI: 0.73–0.74) for gonorrhoea, and 0.65 (95% CI: 0.65–0.66) for chlamydia, which were lower than the original MSHC validation metrics (0.74–0.87, all <i>p</i> &lt; 0.001). Notably, model performance varied across demographic subgroups, with stronger HIV prediction among men who have sex with men with an AUC of 0.78 and better gonorrhoea prediction among younger attendees &lt; 25 years with an AUC of 0.79. At balanced sensitivity-specificity thresholds, the models identified 58.6–64.1% of infections while requiring testing of only 25.8–39.4% of the population.</p> Conclusions <p>Despite performance decrements in external validation using SSHC data, <i>MySTIRisk</i> maintained moderate to good predictive ability across all infections, demonstrating reasonable generalisability across different clinical populations. The demographic variations in performance highlight the importance of context-specific implementation and potential recalibration to optimise clinical utility.</p> Clinical trial number <p>Not applicable.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

External validation of a web- and artificial intelligence-based HIV/STI risk assessment tool: performance evaluation using data from Sydney sexual health centre

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年11月25日

查看全文
  • Phyu Mon Latt,
  • Anik Ray,
  • Heng Lu,
  • Nyi N. Soe,
  • Xianglong Xu,
  • Yining Bao,
  • Jason J. Ong,
  • Eric P. F. Chow,
  • Rick Varma,
  • Lei Zhang,
  • Christopher K. Fairley

摘要

Introduction

HIV and sexually transmitted infections (STIs) continue to pose significant public health challenges globally. MySTIRisk, developed at Melbourne Sexual Health Centre (MSHC), is a machine learning-based tool that predicts individual risk for HIV, syphilis, gonorrhoea, and chlamydia using demographic and behavioural data. While initial validation showed promising results, external validation is crucial to assess its generalisability. This study externally validates MySTIRisk using data from the Sydney Sexual Health Centre (SSHC), Australia’s second-largest sexual health centre.

Methods

Following TRIPOD guidelines, we analysed consultations from patients aged 18 years and older attending SSHC between January 2013 and December 2023. Pre-trained MySTIRisk models were applied directly without modification. Performance was evaluated using the area under the receiver operating characteristic curve (AUC), sensitivity, and specificity at multiple thresholds, with subgroup analyses across demographic characteristics.

Results

We analysed 159,043 to 207,582 consultations at SSHC, depending on the specific infections tested. The median age was 30 years, and 60.2% to 68.8% of the consultations involved men who have sex with men. The area under the receiver operating characteristic curve (AUC) values using data from SSHC were 0.67 (95% CI: 0.65–0.68) for HIV, 0.70 (95% CI: 0.69–0.71) for syphilis, 0.73 (95% CI: 0.73–0.74) for gonorrhoea, and 0.65 (95% CI: 0.65–0.66) for chlamydia, which were lower than the original MSHC validation metrics (0.74–0.87, all p < 0.001). Notably, model performance varied across demographic subgroups, with stronger HIV prediction among men who have sex with men with an AUC of 0.78 and better gonorrhoea prediction among younger attendees < 25 years with an AUC of 0.79. At balanced sensitivity-specificity thresholds, the models identified 58.6–64.1% of infections while requiring testing of only 25.8–39.4% of the population.

Conclusions

Despite performance decrements in external validation using SSHC data, MySTIRisk maintained moderate to good predictive ability across all infections, demonstrating reasonable generalisability across different clinical populations. The demographic variations in performance highlight the importance of context-specific implementation and potential recalibration to optimise clinical utility.

Clinical trial number

Not applicable.