To solve the problems of large design margins and difficult matching between parameter configuration calculation and equipment layout process of auxiliary process system in nuclear power plant, a novel collaborative automation design method was developed in this study, which can be used to determine the system configuration and layout scheme in the overall design stage. Firstly, the relationship and influence between parameter configuration and equipment layout process were analyzed, and a pipeline planning method was established using the A* algorithm to realize the coupling of the design process. Based on the guiding ideology of pursuing better performance in terms of benefits as much as possible while meeting the safety and design requirements, the performance index evaluation model was established as well. Then, based on the basic principle of optimal design, the multi-objective hybrid genetic algorithm and analysis program were developed, so that the matching system design scheme can be optimized. Finally, HPR1000 was taken as the research object. The optimal scheme matching the parameter configuration and equipment layout of the auxiliary process system in nuclear power plant was obtained with the form of program calculation as the main and personnel decision as the auxiliary. This study can effectively shorten the design time, improve the accuracy, avoid the excessive dependence of the engineering experience, and provide a design reference for designers.