Marine Small Modular Reactors (MSMR) have become an important direction in the future development of nuclear energy due to their unique advantages such as high flexibility, ease of deployment, and strong adaptability. However, integrating nuclear systems into compact environments presents significant challenges due to limited space, complex structures, and the immense workload and difficulty of tasks like outfitting, non-destructive testing, and welding. This study presents a novel modularization method for nuclear systems that combines the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm and the Particle Swarm Optimization (PSO) algorithm. Firstly, a correlation matrix is established, encompassing the functional, connection, surface radiation dose, and other relationships, among the components of the nuclear system. Then, the PSO algorithm is utilized to perform multi-objective optimization on the neighborhood radius and minimum density threshold parameters of DBSCAN, achieving adaptive module division. This approach maximizes the correlation of equipment and pipelines within modules while minimizing the inter-module correlations. The effectiveness of this method was demonstrated through a case study of a waste treatment system, showing significant improvements in spatial layout optimization and maintenance accessibility. Results showed that the DBSCAN-PSO method achieved a 34.48% reduction in coupling while maintaining high cohesion, compared to fuzzy hierarchical clustering with improved genetic algorithm. This study provides a new technical solution for the modular design and manufacturing of MSMR and offers a reference for the development and design of small reactors in compact spaces.