Continuous and accurate respiratory monitoring is important for early disease detection and health management. This paper presents a novel multimodal data acquisition system that simultaneously collects Inertial Measurement Unit (IMU) and A-mode ultrasound signals from the chest and abdomen to monitor respiratory activity. We systematically evaluate various classification methods and fusion strategies, including feature-level vector concatenation, tensor fusion, a custom IMU-Ultrasound Convolutional Neural Network with Attention Fusion (IU-CNN-AF), decision-level max fusion, and weighted fusion, both single-subject and cross-subject respiratory state recognition across three breathing patterns (normal, deep, and high-frequency). Experiments on data from nine healthy male volunteers, using Leave-One-Group-Out cross-validation, demonstrate that multimodal fusion significantly outperforms corresponding single-modal methods, especially in more challenging cross-individual scenarios, with decision-level max fusion achieving an accuracy rate of 89.02%, outperforming other methods. Although the available dataset size limits the performance of the IU-CNN-AF network, it still demonstrates potential. These findings highlight the effectiveness and robustness of multimodal sensor fusion for wearable respiratory monitoring and provide valuable insights for future development of portable healthcare systems.