Typhoons are one of the most serious natural disasters affecting the Northwest Pacific Ocean and the coastal areas of China, and their path prediction is highly nonlinear and uncertain due to the influence of multiple complex factors such as atmospheric circulation, topography, and SST. Although the heterogeneous data from multiple sources contains rich information, the existing models are challenging to integrate effectively, and ignoring the deep correlation between different data sources limits the accuracy of typhoon path prediction. To address this challenge, this paper proposes a lightweight prediction model based on a multi-source asymmetric fusion network (MSAFN). The model uses a two-branch LSTM as the backbone, and introduces a multilayer perceptron (MLP) encoder and a three-dimensional convolutional neural network (3D-CNN) for deep feature extraction from multi-source heterogeneous data. The extracted features are fed into the low-order feature interaction and high-order nonlinear interaction modules to achieve the hierarchical fusion of 2D and 3D features. This significantly improves the integration ability of complex influences and the nonlinear evolution’s modeling effect. The experimental results show that MSAFN improves the training efficiency while enhancing the prediction accuracy, which provides a new idea for the intelligent prediction of typhoon paths and the fusion of multi-source meteorological data.