<p>Existing video transformers divide the video frames into a long sequence of tokens and perform self-attention computation among all tokens. However, the tokens corresponding to background information have little effect, and complete utilization of these tokens would generate a lot of computational redundancy. Based on this observation, we design an efficient video transformer named TDKE, which adaptively discards unimportant tokens and solely utilizes important tokens during the inference process. Specifically, the backbone of the TDKE is a 12-layer video transformer, which can be divided into two main parts. The first part, named the Scanner module, is composed of the first two transformer layers. The second part, named the Delicacy module, is composed of the remaining ten transformer layers. The Scanner module quickly discards unimportant tokens of each frame and selects a keyframe. Among them, the importance of each token is measured by pre-calculated attention maps, and the frame with the highest importance score is defined as the keyframe. The Delicacy module uses a weight enhancement technology to the keyframe and further discards redundant tokens based on our enhanced attention maps. We evaluate TDKE on two action recognition datasets, Kinetics-400 and SSv2. The results confirm that TDKE is efficient. For example, TDKE achieved a Top-1 accuracy of 77.8% on the Kinetics-400 dataset using only 306 GFLOPs.</p>