Ventricular septal defects (VSD) can be more effectively identified by combining anatomical structural features from 2D grayscale images and blood flow information from Doppler images. Most current algorithms only perform the same operation on features and do not take into account that different frequency features focus on the expression of different information. Starting from the perspective of frequency and multi-modality, this paper designs a method called multimodality frequency feature customized learning (MFCL) for the identification of VSD. Specifically, this paper first constructs a frequency decomposition module (FD) based on the Fourier transform to extract high-frequency and low-frequency features corresponding to different modes. Secondly, this paper designs a cross-pooling fusion Transformer (CPFT) module tailored for multimodality low-frequency features, which can achieve the fusion of multi-modality global information while reducing computational costs. This paper designs a cross-convolution fusion (CCF) module tailored for multi-modality high-frequency features to achieve the fusion of anatomical structure detail features of 2D grayscale images and blood flow information of color Doppler images. Experimental results show that the proposed algorithm is superior to the comparison methods for VSD identification.