One of the Transformer’s weaknesses is its ignorance of the structural information in the input sentence. Several methods have been proposed to enhance the Transformer’s structural awareness by integrating grammatical structures, such as the dependency tree, into the self-attention module. However, previous methods often have a one-way interaction problem in which the learned information only flows from the relation embedding to the word embeddings during a forward pass. We propose to solve this one-way interaction problem by treating the relation labels in the dependency tree like the words in the sentences. We present the edge relation labels as independent tokens in the input sequence. Then, we could utilize the attention masking mechanism to inform the Transformer model about the structural information by only allowing attention calculations between some of the words and relations. Experiments showed that our approach could improve language models’ performances in various downstream fine-tuning tasks.