For architectural design to actualize climate adaptation, it is essential to optimize building energy efficiency, emission reduction, and passive survivability. However, passive design strategies for building retrofit in the hot summer and cold winter (HSCW) zone are limited in current building energy simulation and optimization (BESO) studies, which have not been widely applied in architectural practice due to the lack of a unified standard. This paper aims to explore the effectiveness and optimization methods of passive design for the typical high-rise apartment retrofit in Philadelphia, considering the dynamic effects of energy consumption, thermal comfort, and future climate scenarios. In this study, the developed future weather files were used to plot the Givoni bioclimatic chart (GBC), and building datasets were constructed based on the EnergyPlus model simulation. Meanwhile, the optimal solutions are realized based on the Morris sensitivity analysis (SA) and NSGA-II method. The results indicate solar protection remains the most effective passive design strategy, especially for south-facing room units, while the cooling effect of natural ventilation by window opening will significantly decrease over time. It is expected that in the future, the thermal coefficient (TC) of the wall and window will increase the effectiveness of energy efficiency to 235% and 152% respectively. The combinations of passive parameters in various climatic scenarios for the overall high-rise apartment retrofit can reduce both heating and cooling loads by up to 50%, and improve the duration of passive survivability by over 400 h.