The Vision Transformer (ViT) has gained widespread adoption in the computer vision domain. In this paper, we propose a smart modified vision transformer module named E-Transformer with extra distinctive tokens which come from high-level feature maps. Specifically, a lightweight squeeze-and-excitation feedforward (SEFF) is introduced in E-Transformer with augmented key and value embeddings for corresponding self-attention generation. We further develop a novel deep neural network with E-Transformer modules (SET) for versatile visual detection and segmentation tasks. Experimental results demonstrate that SET with 1.8 M parameters achieves an accuracy of 98.3% and an inference time of 22.7 ms on the Cornell dataset. Additionally, SET attains impressive performance with the maxFβ of 0.853 and the Sm of 0.854 on the DUTS dataset. As a backbone, SET outperforms several common backbone networks on the VOC07 + 12 dataset, achieves the improvement of 2.2% in mIoU.