The unmanned underwater vehicle (UUV) based on nuclear power has the advantages of high energy density and long endurance. However, the safety of decay heat after reactor shutdown is a major challenge. This paper presents a new reactor residual heat removal system which combines direct heat conduction in reactor vessel with variable conductance heat pipes. When the reactor is shut down, some of the heat is lost through the reactor vessel, while the remaining heat is transported from the reactor core to the waste heat exchanger through the variable conductance heat pipes. The effects of different VCHP arrangements and different thermal fillers on the residual heat removal system were analyzed by CFD method, and the optimal design scheme of the residual heat removal system was obtained. The calculation results show that the maximum core temperature is 1169.8 K and the maximum UUV housing temperature is 330.85 K, which meets the safety limit requirements. The above research proves the feasibility of the residual heat removal method based on VCHP.