With the rapid growth of scientific paper data, intelligent fine-grained document classification techniques can significantly benefit paper retrieval and recommendation. Previous works on paper classification focused on texts and were coarse-grained. The high sample similarity within subdivided fields makes fine-grained classification more challenging. Given the implicit semantic relationship between abstracts and figures in academic papers, current multimodal methods designed to process explicitly related textual and visual data are inadequate for addressing the fine-grained multimodal paper classification task. Additionally, discrepancies in data distribution and dataset limitations frequently result in overfitting when pretrained models are transferred to paper classification tasks. In this work, we propose a multimodal regularization ensemble network for fine-grained paper classification. Our model focuses on the differentiation of intra-modality feature representations and the enhancement of inter-modality related information for the task. Comparative analysis with baseline methods highlights the efficiency of our proposed model. Finally, we developed a paper classification system based on our model that is capable of classifying, managing, and recommending academic papers.