Scholarlink

Background <p>Interrupted Time Series (ITS) analysis is increasingly used to evaluate healthcare quality improvement (QI) interventions, yet limited guidance exists on which analytical approach performs best under the complex data conditions typical of QI research.</p> Methods <p>A Monte Carlo simulation study was conducted using a full factorial design with 512 unique scenarios (varying time series length, autocorrelation, trend type, seasonality, noise level, intervention effects, external shocks, and covariates), each replicated 500 times, yielding 256,000 simulated datasets. Four models were compared: ARIMAX, Generalized Additive Models (GAM), Prophet, and Long Short-Term Memory (LSTM) networks. Model performance was evaluated using bias, root mean squared error (RMSE), statistical power, and Type I error rate. For counterfactual-based models (Prophet and LSTM) lacking traditional <i>p</i>-values, an empirical null distribution approach was used to derive comparable significance tests.</p> Results <p>ARIMAX demonstrated the highest estimation accuracy (RMSE = 7.85) and statistical power for detecting level changes (72.3%), followed by GAM (RMSE = 8.69; power = 70.2%). However, both parametric models exhibited inflated Type I error rates (ARIMAX: 13.8%; GAM: 9.1% for level change), particularly pronounced for slope change detection (ARIMAX: 37.2%). In contrast, Prophet maintained well-controlled Type I error (4.8%) but with substantially lower power (15.4%) and higher estimation error (RMSE = 20.1). LSTM achieved 25.4% power and an RMSE of 14.8 among the 35.5% of simulated datasets in which the model converged; LSTM failed to converge in 64.5% of simulated datasets, predominantly in short series (T = 36&#xa0;months). Seasonality was the dominant source of estimation bias for parametric models, while trend type was the primary driver for counterfactual models. Model performance degraded progressively with scenario complexity, with power declining from near 100% (simplest conditions) to approximately 50% (most complex conditions) for ARIMAX and GAM.</p> Conclusions <p>No single method dominates across all performance criteria. ARIMAX offers the best accuracy and power but risks false positives, particularly for slope change effects. GAM provides a balanced alternative with better Type I error control and robustness to nonlinear trends. Prophet and LSTM are conservative approaches suitable when avoiding false positive conclusions is paramount. These findings provide empirical guidance for selecting ITS analytical methods appropriate to specific QI research contexts and priorities.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Comparative evaluation of interrupted time series analytical methods for healthcare quality improvement research: a Monte Carlo simulation study

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月26日

查看全文
  • Seungman Kim,
  • Jaehoon Lee

摘要

Background

Interrupted Time Series (ITS) analysis is increasingly used to evaluate healthcare quality improvement (QI) interventions, yet limited guidance exists on which analytical approach performs best under the complex data conditions typical of QI research.

Methods

A Monte Carlo simulation study was conducted using a full factorial design with 512 unique scenarios (varying time series length, autocorrelation, trend type, seasonality, noise level, intervention effects, external shocks, and covariates), each replicated 500 times, yielding 256,000 simulated datasets. Four models were compared: ARIMAX, Generalized Additive Models (GAM), Prophet, and Long Short-Term Memory (LSTM) networks. Model performance was evaluated using bias, root mean squared error (RMSE), statistical power, and Type I error rate. For counterfactual-based models (Prophet and LSTM) lacking traditional p-values, an empirical null distribution approach was used to derive comparable significance tests.

Results

ARIMAX demonstrated the highest estimation accuracy (RMSE = 7.85) and statistical power for detecting level changes (72.3%), followed by GAM (RMSE = 8.69; power = 70.2%). However, both parametric models exhibited inflated Type I error rates (ARIMAX: 13.8%; GAM: 9.1% for level change), particularly pronounced for slope change detection (ARIMAX: 37.2%). In contrast, Prophet maintained well-controlled Type I error (4.8%) but with substantially lower power (15.4%) and higher estimation error (RMSE = 20.1). LSTM achieved 25.4% power and an RMSE of 14.8 among the 35.5% of simulated datasets in which the model converged; LSTM failed to converge in 64.5% of simulated datasets, predominantly in short series (T = 36 months). Seasonality was the dominant source of estimation bias for parametric models, while trend type was the primary driver for counterfactual models. Model performance degraded progressively with scenario complexity, with power declining from near 100% (simplest conditions) to approximately 50% (most complex conditions) for ARIMAX and GAM.

Conclusions

No single method dominates across all performance criteria. ARIMAX offers the best accuracy and power but risks false positives, particularly for slope change effects. GAM provides a balanced alternative with better Type I error control and robustness to nonlinear trends. Prophet and LSTM are conservative approaches suitable when avoiding false positive conclusions is paramount. These findings provide empirical guidance for selecting ITS analytical methods appropriate to specific QI research contexts and priorities.