Gesture recognition using smartphones is crucial for enabling intuitive human-computer interaction, particularly under real-world conditions. However, reliable gesture recognition across diverse users while performing physical activities, such as walking, remains challenging. Consequently, this paper presents a person-independent gesture recognition system that utilizes gravitational readings derived from the smartphone’s IMU sensor for robust smartphone-based gesture recognition. The system was evaluated based on data from 13 participants who performed four different gestures while walking or standing. We have trained and evaluated a Support Vector Machine (SVM) and a Random Forest (RF) classifier to distinguish five classes and achieved an accuracy of \(91.37\%\) for Support Vector Machine, and \(90.31\%\) for Random Forest over all participants and gestures. Furthermore, we applied leave-one-subject-out to prove person independence (mean accuracy \(88.25\%\) SVM; \(87.53\%\) RF), in addition to robustness against walking (mean accuracy \(83.64\%\) SVM; \(81.21\%\) RF). The results indicate the system’s suitability for real-world, smartphone-based gesture recognition.