In seismically active metropolitan areas, significant attention has been paid to the vulnerability of super high-rise buildings to earthquakes. In recent decades, Japan has experienced numerous major earthquakes, during which damage to super high-rise buildings due to long-period and long-duration ground motions has become a significant concern. For example, during the 2011 Great East Japan Earthquake, the 54-story Sakishima Government Office Building in Osaka experienced a displacement of approximately 1.3 m, with the shaking lasting for more than 10 min. Although visual inspection revealed damage to non-structural infill walls, there was debate on whether the building should be demolished or strengthened for continued use. The loss of functionality in super high-rise buildings significantly impacts community resilience. Therefore, it is crucial to conduct a detailed damage assessment and develop effective detection systems, such as Structural Health Monitoring (SHM), for these buildings. Traditionally, horizontal vibration observation data have been used to examine only properties related to shear stiffness. In this study, to enhance the efficiency of damage identification in super high-rise buildings using observational data, we analyze both shear and bending deformation properties—specifically, shear-wave velocity (Cs) and longitudinal-wave velocity (Cl)—by modeling the building as a Timoshenko beam. If structural damage occurs, both Cs and Cl decrease. The identification of Cs and Cl is performed by fitting impulse response data with respect to a virtual source at the base using a Data-Driven Neural Network (DDNN). Furthermore, to achieve highly reliable estimates of Cs and Cl, a decoupled Timoshenko beam equation—incorporating only horizontal displacement—is also incorporated into the DDNN as physical information, thereby forming a Physics-Informed Neural Network (PINN).