<p>Real-time Isolated Sign Language Recognition (ISLR) requires high accuracy and low latency. These competing demands continue to pose challenges for existing frame-based and transformer-heavy architectures. To mitigate this, we develop a Real-Time Spatio-Temporal Adaptive Motion Pattern framework (RT-STAMP-SLR) for event-driven recognition of isolated sign language gestures. RT-STAMP-SLR is different from other systems since it does not process all the video frames. Instead, it uses a motion-based frame selection technique that reduces unnecessary calculations by processing frames only when there is a meaningful movement of the hand or body. The redundancy and computational costs are significantly lower. The frames selected are transformed into compact gesture tokens that represent the hand’s shape, motion, and spatial movement. These tokens are then analyzed by a temporal memory transformer, which compresses the signs by retaining only the relevant motion patterns and eliminating the redundant ones. An early decision mechanism is employed to provide predictions before the gesture is completed, thereby offering low latency. The proposed Amrita Real-Time SLR Dataset includes a wide range of dynamic gesture movements recorded under real-time streaming conditions. Extensive evaluations demonstrate that the RT-STAMP-SLR system is well-suited for assistive and human–computer interaction applications, even under practical real-time constraints.</p>