Attention Is a Functor: Enforcing Categorical Structure in Transformers
摘要
We present the world’s first transformer model that treats attention heads as functors in the categorical sense. By modeling tokens as objects and attention links as morphisms, our architecture enforces identity and composition laws from category theory directly in the attention mechanism. This is achieved through a novel functor loss that penalizes violations of structure-preserving constraints. The resulting Functor Attention mechanism improves compositional generalization, enhances interpretability, and induces more stable token graphs. We evaluate our model on SCAN and GeoQuery, two benchmarks requiring strong compositional reasoning, and show significant improvements over standard transformers. Our work opens a new direction for grounding deep learning models in mathematical structures with provable properties. The complete implementation is available at https://github.com/satyamcser/attention-as-functor-transformer .