We present the world’s first transformer model that treats attention heads as functors in the categorical sense. By modeling tokens as objects and attention links as morphisms, our architecture enforces identity and composition laws from category theory directly in the attention mechanism. This is achieved through a novel functor loss that penalizes violations of structure-preserving constraints. The resulting Functor Attention mechanism improves compositional generalization, enhances interpretability, and induces more stable token graphs. We evaluate our model on SCAN and GeoQuery, two benchmarks requiring strong compositional reasoning, and show significant improvements over standard transformers. Our work opens a new direction for grounding deep learning models in mathematical structures with provable properties. The complete implementation is available at https://github.com/satyamcser/attention-as-functor-transformer .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Attention Is a Functor: Enforcing Categorical Structure in Transformers

  • Hyun Kim,
  • Satyam Mishra,
  • Tran Duc Tan,
  • Vishwanath Bijalwan

摘要

We present the world’s first transformer model that treats attention heads as functors in the categorical sense. By modeling tokens as objects and attention links as morphisms, our architecture enforces identity and composition laws from category theory directly in the attention mechanism. This is achieved through a novel functor loss that penalizes violations of structure-preserving constraints. The resulting Functor Attention mechanism improves compositional generalization, enhances interpretability, and induces more stable token graphs. We evaluate our model on SCAN and GeoQuery, two benchmarks requiring strong compositional reasoning, and show significant improvements over standard transformers. Our work opens a new direction for grounding deep learning models in mathematical structures with provable properties. The complete implementation is available at https://github.com/satyamcser/attention-as-functor-transformer .