<p>In recent years, Audio-Visual Emotion Recognition (AVER) has gained increasing attention due to its immense potential in human-computer interaction and multimedia applications. Despite significant progress, effective audio-visual emotion recognition still faces challenges, including difficulty in synergizing time-frequency evidence, weak cross-modal consistency and underutilization of complementary cues, as well as redundancy-induced interference. To address these issues, we propose a novel framework for intra-modal and inter-modal interactions (FMII). At the intra-modal level, we design a time-frequency adaptive fusion strategy, which enhances audio representations’ robustness by learning bidirectional time-frequency mappings and applying soft masks to filter key segments, along with domain-level adaptive re-scaling. At the inter-modal level, we first address the lack of consistency in modality-invariant representations by developing a hierarchical multi-granularity dynamic cross-modal attention fusion strategy. This strategy captures and balances global semantic cues and local details, improving the discriminability and robustness of the fused representation. Additionally, to mitigate redundant interference in modality-specific representations, we introduce a feature-level complementary information disentanglement strategy. By maximizing discriminative mutual information and reducing cross-subspace interference, we decompose each modality’s features into task-relevant and task-irrelevant subspaces, preserving useful complementary cues while filtering out redundant information. These combined strategies enable FMII to effectively capture intra-modal information, inter-modal consistency, and complementary cues, while minimizing noise, resulting in robust unified emotion representations. Comprehensive experiments on three popular datasets (RAVDESS, CREMA-D, and CMU-MOSEI) demonstrate that our method achieves superior performance on the AVER task.</p>