Temporal Forgery Localization (TFL) aims to localize tampered segments within locally forged media precisely. Existing methods improve localization accuracy by combining multi-modal information and embedding multi-scale features. However, they often prioritize the intra-modal information and overlook the significance of inter-modal information, such as insufficient alignment between modalities and failure in leveraging latent inter-modal specifics. To address these limitations, we propose a cross-modal framework that captures deep inter-modal information through enhanced feature alignment and fusion. Specifically, through cross-modal reconstruction, we design a Bidirectional Reconstruction Module (BRM) to achieve comprehensive two-stage alignment and capture inter-modal correspondence. This shifts the uni-modal detection perspective to the other modality for more effective detection. Further, we introduce a Dynamic Feature Fusion Module (DFFM) to exploit latent specifics like temporal inconsistencies and data distribution flaws for feature enhancement. Extensive experimental results obtained on the Lav-DF benchmark demonstrate the superiority of the proposed method over the existing approaches, particularly in high-precision localization. Moreover, the proposed modules excel in fake news detection and grounding, validating their robustness and generalization capability for fine-grained multi-modal partial forgery detection tasks. The source code of the proposed method is available at https://github.com/garlicyh/CIMA .