Human Activity Recognition systems have applications in various domains ranging from health and fitness to medical applications. Many of the existing work focuses only on model building and benchmarking. However, in this study, we aimed to design and develop an effective human activity recognition system that is explainable and deployable in a real-time setting. Our work used the capability of joint learning algorithms like the joint multi-head memory models to classify complex human activity recognition tasks. We designed and developed the Joint Convolutional Neural Networks and Long Short Term Memory model that achieved an accuracy of 99.03% on the UCI human activity recognition dataset. We also focussed on model and data explainability using SHapley Additive exPlanations and Local Interpretable Model-agnostic Explanations algorithms. The explainability module highlights the top 15 features that are critical in the model’s thinking and decision-making. Finally, we developed an android application for real-world human activity recognition based on phone sensor data that achieved an accuracy of 91.75%.