<p>Human language processing, characterized by its ability to capture long-range dependencies in sequential inputs, operates under the constraints of limited working memory. In contrast, state-of-the-art transformer models in artificial intelligence rely on access to the fixed context window, which deviates from the dynamic nature of human cognition. Here, we propose a novel approach to reconcile this disparity by integrating a computational model of working memory into the transformer architecture. This biologically-inspired modification constructs a time-local transformer, capable of learning complex dependencies without needing the full input history. Our findings demonstrate that this approach still preserves the capacity of transformers for effective sequence processing. This work is a step towards developing AI models that align more closely with the principles of human brain function, opening new avenues for understanding the neural underpinnings of language and cognition.</p>