The rise of vast, unedited video content has made it crucial to find clips that match text queries accurately. The existing methods typically create clips to match the text queries by leveraging sliding windows or uniform sampling. However, these methods usually lead to inefficient and inaccurate retrieval due to their failure to effectively capture the interaction between query and video content at a fine-grained level, and they struggle with scalability when dealing with large-scale video datasets. To address these issues, we propose a novel Frame-Wise Multimodal Retrieval framework with contrastive learning for video moment retrieval (FCVR). Firstly, FCVR independently encodes text and video with unimodal encoding model, respectively. Secondly, a frame-level contrastive learning module and a video-level contrastive learning module are designed for further improving the efficiency and precision of video moment retrieval. Specifically, an internal-frame prediction module is designed to evaluate the similarity between frames by using the frame similarity score module, which significantly enhance the ability of locate video content related to text queries via fine-grained frame-level analysis. Extensive experiments demonstrate the superiority of FCVR over several state-of-the-art methods in terms of both accuracy and retrieval efficiency.