<p>Video temporal sentence localization aims to localize a target moment in videos given language queries. We observe that existing models suffer from a sheer performance drop when dealing with phrases contained in the sentence. It reveals the limitation that existing models lack sufficient understanding of the semantic phrases in the query. To address this problem, we fully exploit the temporal constraints between phrases within the same sentence and attempt to transfer knowledge from externally pre-trained large models to help the model better accomplish phrase-level localization. Firstly, we propose a phrase-level Temporal Relationship Mining (TRM) framework that employs the temporal relationship between the phrase and the whole sentence to better understand each semantic entity (e.g. verb, subject) in the sentence. Specifically, we propose the consistency and exclusiveness constraints between phrase and sentence predictions to improve phrase-level prediction quality and use phrase-level predictions to refine sentence-level ones. Then, we extend the TRM framework with phrase-level training (TRM-PT) using the large-scale pre-trained models to generate fine-grained pseudo-labels for the phrase. To mitigate the negative impact of the label noise, we further propose to iteratively optimize the pseudo-labels. Finally, to enhance the understanding of verb phrases, we utilize a language model to infer changes in the scene’s state before and after the occurrence of verb phrases and align them with the visual content. Experiments on the ActivityNet Captions and Charades-STA datasets show the effectiveness of our method on both phrase and sentence temporal localization and enable better model interpretability and generalization when dealing with unseen compositions of seen concepts. The code is available at <a href="https://github.com/minghangz/trm">https://github.com/minghangz/trm</a>.</p>