Inertial Measurement Unit (IMU)-based Human Activity Recognition (HAR) systems that employ Generalized Zero-Shot Learning (GZSL) face significant challenges in accurately classifying activities that were not observed previously during training. These challenges stem primarily from the inherent difficulty of recognizing unseen classes without sacrificing the classification accuracy of observed classes in a GZSL setting. A novel deep neural network (DNN) architecture termed as the Joint Sequences (JS)-Siamese architecture is proposed to address these challenges using IMU and video data. The proposed architecture uses skeleton joint sequences to bridge the gap between IMU features and video data, thus effectively solving the domain shift problem. A Siamese DNN-based metric learning model is employed to handle the hubness problem by mapping similar samples in close proximity and dissimilar ones farther apart in a joint embedding space. Additionally, a Dynamic Calibration Ensemble (DCE) technique is introduced to address the classification bias towards the observed classes in GZSL, thereby ensuring balanced representation of both, observed and unseen classes. The proposed JS-Siamese DNN architecture is shown to yield significant performance improvement over attribute-based, word embedding-based and video embedding-based GZSL approaches for HAR proposed in the literature. Experimental evaluation on three IMU benchmark datasets, i.e., PAMAP2, DaLiAc and UTD-MHAD demonstrate the effectiveness of the proposed JS-Siamese DNN architecture for sensor-based HAR.