Deep neural networks (DNNs) have emerged as a powerful tool for solving regression problems, offering flexible architectures that can be tailored to diverse applications. In earthquake engineering, accurately predicting the seismic response of buildings remains a critical challenge. While traditional simplified models of single- or multi- degree-of-freedom systems, are widely used due to their computational efficiency and ability to facilitate rapid simulations, these approaches often fall short in capturing complex nonlinear structural behavior and spatial variability in ground motions. This research integrates earthquake time-histories with ambient vibration oscillators’ data to develop an advanced neural network-based predictive model. Presented through image representations, this framework is meticulously designed to enhance the accuracy of forecasting structural seismic responses taking also into account stiffness non-linearity. The goal is to accurately forecast a building’s seismic response. A dataset comprising 1197 MDOF 2D models was utilized, producing a total of 32,319 training samples for the model. The proposed framework is evaluated using except the loss function of a network model training, also with a mean absolute percentage error (MAPE). By combining AV and EQ response data into a neural network-based approach, the proposed network demonstrates a new direction developing tools using neural networks for predicting the seismic response of structures. Such advancements could not only enhance the understanding of building behavior during earthquakes but also support the development of resilient building designs, contributing to safer engineering practices and improved earthquake mitigation strategies.