Scholarlink

<p>The modeling of high-dimensional spatio-temporal processes presents a fundamental dichotomy between the probabilistic rigor of classical geostatistics and the flexible, high-capacity representations of deep learning. While Gaussian processes offer theoretical consistency and exact uncertainty quantification, their prohibitive computational scaling renders them impractical for massive sensor networks. Conversely, modern transformer architectures excel at sequence modeling but inherently lack a native geometric understanding, treating spatial sensors as permutation-equivariant tokens without a native understanding of distance. In this work, we propose a Spatially-Informed Transformer, a hybrid architecture that injects a geostatistical inductive bias directly into the self-attention mechanism via a learnable covariance kernel. By formally decomposing the attention structure into a stationary geostatistical structure and a non-stationary data-driven residual, we impose a soft metric constraint that favors spatially proximal interactions while retaining the capacity to model complex dynamics. We demonstrate the phenomenon of “Deep Variography”, where the network successfully recovers the true spatial decay parameters of the underlying process end-to-end via backpropagation. Extensive simulation studies on synthetic Gaussian Random Fields confirm that our method accurately recovers the theoretical covariance structure and provides well-calibrated probabilistic forecasts, effectively bridging the gap between interpretable spatial statistics and data-driven learning.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Spatially-informed transformers: injecting geostatistical covariance structure into self-attention for spatio-temporal forecasting

  • 期刊论文

  • 发表日期：

    2026年4月10日

查看全文
  • Yuri Calleo

摘要

The modeling of high-dimensional spatio-temporal processes presents a fundamental dichotomy between the probabilistic rigor of classical geostatistics and the flexible, high-capacity representations of deep learning. While Gaussian processes offer theoretical consistency and exact uncertainty quantification, their prohibitive computational scaling renders them impractical for massive sensor networks. Conversely, modern transformer architectures excel at sequence modeling but inherently lack a native geometric understanding, treating spatial sensors as permutation-equivariant tokens without a native understanding of distance. In this work, we propose a Spatially-Informed Transformer, a hybrid architecture that injects a geostatistical inductive bias directly into the self-attention mechanism via a learnable covariance kernel. By formally decomposing the attention structure into a stationary geostatistical structure and a non-stationary data-driven residual, we impose a soft metric constraint that favors spatially proximal interactions while retaining the capacity to model complex dynamics. We demonstrate the phenomenon of “Deep Variography”, where the network successfully recovers the true spatial decay parameters of the underlying process end-to-end via backpropagation. Extensive simulation studies on synthetic Gaussian Random Fields confirm that our method accurately recovers the theoretical covariance structure and provides well-calibrated probabilistic forecasts, effectively bridging the gap between interpretable spatial statistics and data-driven learning.