Inductive biases are essential for efficient inductive learning. In neural networks, convolutional layers are biased towards learning location-invariant features, whereas attention layers excel at capturing dependencies in sequential data. Here, we systematically study and unify the biases of these architectures within the framework of geometric deep learning to reveal their underlying similarities. We demonstrate that local attention layers exhibit a relational inductive bias similar to that of convolutional layers. Based on this finding, we construct a local attention layer for image processing that assumes a grid structure but models the data dynamically and stochastically, in contrast to conventional convolutional layers, which use static and deterministic assumptions.