Self-attention-based Transformers play a significant role in computer vision. Adapting the Transformer architecture from language to vision encounters challenges arising from domain-specific differences. Notably, the translation equivariance of objects in images presents a distinct contrast to the linguistic context of words in text. Such differences are especially evident in downstream vision tasks, especially object detection. In this paper, we propose a novel symmetric attention mechanism called Symmetric Kernel Attention (SKA). In SKA, tokens are directly used for symmetric kernel attention computation rather than mapped into QKV space such that attention weights are invariant from different token orders. Besides, similarity among tokens is measured with our proposed RBF kernel computation algorithm, in which tokens are implicitly projected into an infinite dimensional feature space. Meanwhile, in order to improve Transformer block’s representation power, we combine Kernel Attention with Channel Attention to build a new type of Transformer block. These new Transformer blocks not only learn semantic information from pixels, but also focus on channel-wise feature relations. Several Transformer blocks consist of a hierarchical structure of the feature extraction network called Symmetric Kernel Attention Network, which serves as the backbone for object detection frameworks. We conduct experiments on the MS-COCO and Pascal VOC and train from scratch. Results show that our method outperforms Swin Transformer by 1.1% on MS-COCO and 2.5% on Pascal VOC.