The current multimodal maritime vessel recognition methods suffer from inadequate extraction of cross-modal feature interactions and multi-modal information loss caused by late fusion. To address this issue, a multimodal vessel recognition network, termed as CAMF, is proposed, targeting the visible light and infrared modalities. CAMF employs two encoders with non-shared weights and utilizes a multi-stage fusion approach to achieve a unified representation of the two modalities. During the multi-stage fusion process, a cross-modal interaction module is introduced that computes modality cross-attention and fuses them in the form of low-rank matrices, facilitating the encoders to capture cross-modal interaction features. In the later stages of fusion, to minimize information redundancy and loss, a fusion method is designed based on information entropy and modality confidence. The superiority of our proposed method is validated on public VAIS datasets.