Federated Learning (FL) is crucial for privacy-preserving, cross-institutional medical image segmentation. However, current FL methods mainly handle single modalities, while multimodal imaging offers complementary data for better accuracy. Challenges in fusing diverse modalities, like differing grayscale distributions and texture conflicts, hinder multimodal FL segmentation. This paper introduces a novel cross-modal FL framework, the Dual-Dimensional Optimization Framework. It optimizes federated aggregation by addressing background spectral characteristics and deep semantic feature distributions to reduce inter-modal heterogeneity while ensuring privacy. Key methods include frequency-domain analysis to remove device-specific variations and an anatomical consistency constraint for better data uniformity. An adaptive feature distribution strategy refines aggregation to efficiently extract complementary multimodal information. Our framework leverages multimodal data complementarity, boosts global model generalization, and keeps sensitive image data localized to prevent privacy leaks. Experiments show significantly improved segmentation on four heterogeneous tasks, especially with non-IID data. This work offers an efficient, privacy-preserving solution for multimodal federated medical image segmentation, advancing cross-institutional collaboration.