Scholarlink

Purpose <p>Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success in Natural Language Processing and Computer Vision domains. However, the development of PTMs on healthcare time-series data is lagging behind. This underscores the limitations of the existing transformer-based architectures, particularly their scalability to handle large-scale time series and ability to capture long-term temporal dependencies.</p> Methods <p>In this study, we present Timely Generative Pre-trained Transformer (TimelyGPT). TimelyGPT employs an extrapolatable position (xPos) embedding to encode trend and periodic patterns into time-series representations. It also integrates recurrent attention and temporal convolution modules to effectively capture global–local temporal dependencies.</p> Results <p>Our experiments show that TimelyGPT excels in modeling continuously monitored biosignals and irregularly-sampled time series data commonly observed in longitudinal electronic health records (EHRs). In forecasting continuous biosignals, TimelyGPT achieves accurate extrapolation up to 6000 timesteps of body temperature during the sleep stage transition given a short look-up window (i.e., prompt) containing only 2000 timesteps. For irregularly-sampled time series, TimelyGPT with a proposed time-specific inference demonstrates high top recall scores in predicting future diagnoses using early diagnostic records, effectively handling irregular intervals between clinical records. We further demonstrate that TimelyGPT achieves strong discriminative performance on both continuous and irregularly-sampled time series.</p> Conclusion <p>Together, we envision TimelyGPT to be useful in various health domains, including long-term patient health state forecasting, patient risk trajectory prediction, and disease classification. Its code is available at <a href="https://github.com/li-lab-mcgill/TimelyGPT">Github</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Timelygpt: extrapolatable transformer pre-training for long-term time-series forecasting in healthcare

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年10月14日

查看全文
  • Ziyang Song,
  • Qincheng Lu,
  • Hao Xu,
  • Ziqi Yang,
  • He Zhu,
  • David Buckeridge,
  • Yue Li

摘要

Purpose

Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success in Natural Language Processing and Computer Vision domains. However, the development of PTMs on healthcare time-series data is lagging behind. This underscores the limitations of the existing transformer-based architectures, particularly their scalability to handle large-scale time series and ability to capture long-term temporal dependencies.

Methods

In this study, we present Timely Generative Pre-trained Transformer (TimelyGPT). TimelyGPT employs an extrapolatable position (xPos) embedding to encode trend and periodic patterns into time-series representations. It also integrates recurrent attention and temporal convolution modules to effectively capture global–local temporal dependencies.

Results

Our experiments show that TimelyGPT excels in modeling continuously monitored biosignals and irregularly-sampled time series data commonly observed in longitudinal electronic health records (EHRs). In forecasting continuous biosignals, TimelyGPT achieves accurate extrapolation up to 6000 timesteps of body temperature during the sleep stage transition given a short look-up window (i.e., prompt) containing only 2000 timesteps. For irregularly-sampled time series, TimelyGPT with a proposed time-specific inference demonstrates high top recall scores in predicting future diagnoses using early diagnostic records, effectively handling irregular intervals between clinical records. We further demonstrate that TimelyGPT achieves strong discriminative performance on both continuous and irregularly-sampled time series.

Conclusion

Together, we envision TimelyGPT to be useful in various health domains, including long-term patient health state forecasting, patient risk trajectory prediction, and disease classification. Its code is available at Github.