Natural Language Processing (NLP) has been transformed by transformer-based language models, and attention mechanisms (particularly Multi-Head Attention (MHA)) have drawn a lot of interpretive attention. Other elements of the Transformer encoder block, like Feed-Forward Networks (FFN) and Residual Connections, are still not well understood. In order to separate the syntactic representation contributions of MHA, FFN, and residuals, we decompose and linearize the encoder block in this work. We assess each component’s ability to encode syntactic structure using syntax probing techniques. Our findings demonstrate that although each component makes a contribution, MHA makes a smaller contribution than is generally believed, while Residual Connections carry more syntactic information than anticipated. These results provide fresh perspectives on the internal organization of language models and guide future architectural advancements for enhanced syntactic comprehension