<p>Human action recognition in videos requires extensive computational resources to process frame sequences and extract actionable features. However, most of the processed pixels in each frame are irrelevant to the underlying action and can simply be excluded from processing without hindering the action recognition task. Additionally, achieving real-time action recognition is required by several applications, especially when public safety and crime detection is at hand. Inten- sifying the effort towards recognizing a human action while omitting irrelevant information can greatly expedite performance. In this paper, we propose a novel methodology for training action recognition systems using only human body landmarks. We develop a violence recognition system that utilizes the proposed approach for training and inference. We ana- lyze the impact of different system design choices, namely the sequence length and body landmarks, on the system’s accuracy. We also show that the proposed system interprets body language better with additional landmarks for the face and hands. Moreover, we evaluate the performance of the proposed system on four video datasets. The empirical results show that the proposed system can achieve similar or better accuracy than existing state-of-the-art methods while substantially reducing the model size by <b>62%</b> to <b>85%</b>. Furthermore, we demon- strate the portability of the proposed system by analyzing its performance with respect to variations in model size.</p>