<p>Self-attention-based models are widely used in tasks like image classification, object detection, and video comprehension, which have achieved remarkable success. Inspired by these advancements, we explore the potential of transformer networks for temporal action detection (TAD) in videos. We aim to investigate how transformers can effectively localize actions in an untrimmed video. The solution we propose is DroFormer (drop-based temporal action transformer), a novel one-stage action detector that can accurately identify action boundaries and classify actions. Notably, DroFormer accomplishes this without requiring action proposals or predefined anchor windows. By leveraging the drop mechanism in self-attention and utilizing a lightweight decoder, DroFormer effectively captures the necessary contextual information for understanding actions and classifies each moment in time to estimate the precise action boundaries. Additionally, we introduce the Action Refinement Loss (ARL) to tackle the issue of imbalanced action distributions. ARL incorporates a focal loss function that adjusts the loss weight of positive–negative samples as well as challenging samples. Benefiting from the drop mechanism and ARL, DroFormer achieves 54.98% mAP at tIoU = 0.5 on ActivityNet 1.3, outperforming the prior model by 0.31 absolute percentage points. Moreover, our method reaches state-of-the-art performance on EPIC-Kitchens 100, with a boost of over 1.11 absolute percentage points in average mAP.</p>