<p>Video moment retrieval and highlight detection aim to localize the moment and estimate clip-wise saliency score in a video given a user query. Despite some progress achieved by transformer-based models, we find that they usually fail in moment-level video-text semantic alignment when predicted moments and GT moments are in the intersection situation. In addition, their highlight detection is only for certain clips and does not focus on all clips in the video, thus leading to inaccurate clip-level video-text semantic alignment. To resolve these issues, we design a novel transformer-based model called dual-level semantic alignment DETR. Specifically, we design a novel regression by introducing the center distance called center distance regression to achieve moment-level semantic alignment. Furthermore, we consider all clips in the whole video, and add a saliency contrastive learning module, using representation learning to complete clip-level semantic alignment. Extensive experiments on three benchmarks, i.e., QVHighlights, Charades-STA, and TVSum, show the effectiveness of our approach and the strong competition on the moment retrieval and highlight detection task. Without bells and whistles, we observe a 7.26% average mAP improvement on the QVHighlights benchmark and a gain of 4.1% average score in 10 categories on the TVSum benchmark.</p>