<p>Video moment retrieval (MR) and highlight detection (HD) are two fundamental tasks in multimodal video understanding. They focus on moment localization and key clip selection, respectively, and are widely applied in content recommendation and intelligent retrieval. Although differing in objectives and outputs, both tasks involve vision-language alignment and complex temporal structure modeling, thus possessing natural synergy. However, existing joint modeling approaches often fail to fully exploit structural information in videos, leading to degraded performance in scenarios with ambiguous semantic boundaries or frequent behavioral transitions. To overcome these challenges, we propose a structure-aware joint modeling framework, SE-DETR (Structure-Enhanced DETR), to enhance structural representation and task interaction between MR and HD via Saliency-Guided Structural Retrieval and Boundary-Driven Highlight Re-estimation. SE-DETR reconstructs temporal structure features through a structure-guided module that incorporates slot-based aggregation and boundary response prediction, providing a unified structural prior for downstream tasks. Moreover, a task interaction mechanism is developed, unifying a saliency-guided structural retrieval path with a boundary-driven highlight re-estimation path, thereby enhancing the model’s adaptability to abrupt semantic transitions and improving the precision of target localization. Additionally, a boundary supervision loss based on semantic variation rate is introduced to refine boundary sensitivity without requiring additional annotations, enhancing the model’s ability to represent vague behavior boundaries. Under identical features and evaluation protocols, SE-DETR achieves strong overall performance on QVHighlights, Charades-STA, and TVSum, attaining the highest overall mean AP across these benchmarks compared with recent methods, without requiring extra annotations.</p>