A motor imagery based brain-computer interface (MI-BCI) has found extensive application in the field of neural decoding, primarily focusing on recognizing motor intentions. Current research on MI-BCI is advancing towards decoding Fine Joint Movement Imagery (FJMI), which emphasizes movements of various joints within the same limb. However, the neural activation differences for FJMI are smaller, and the activation regions are closer together, making it more challenging to identify motor imagery (MI) with low accuracy. How to further improve the accuracy of FJMI decoding remains to be studied. Therefore, the purpose of this paper is to enhance the performance of FJMI decoding by integrating the feature data from two modalities, EEG and fNIRS. We proposed a novel multimodal fusion method named Filter Bank Sinc-Convolutional Fusion Network (FB-Sinc-FusionNet) and combined it with a new two-stage training strategy to address the 4-class FJMI problem. Comparative experiments on the data collected from 18 subjects demonstrated that our proposed method exceeded the performance of typical deep learning models, reaching an accuracy of 65.65% in the four-category classification. This paper exhibits the potential and advantages of multi-modality signal fusion decoding to improve the efficiency of FJMI decoding.