This research addresses the critical need for accurate sign language recognition systems, specifically targeting New Zealand Sign Language (NZSL), one of three official languages in New Zealand. Despite its importance in serving a significant portion of the population, existing systems struggle with accurately interpreting dynamic signs, especially under varying conditions and levels of occlusion. To overcome these challenges, we leveraged deep learning and innovative data collection methodologies, creating a dedicated environment for data collection and developing a tailored pipeline to process the data. The primary challenge was accurately capturing the positions of multiple points on a person’s hands within an augmented reality space using the Leap Motion controller and effectively extracting this data for processing. The experiments demonstrated that the proposed deep learning pipeline significantly improves the recognition accuracy of dynamic gestures. Specifically, the model utilizing the Adam optimizer outperformed those using RMSProp and Adagrad in terms of precision, AUC-ROC, and log loss. The impact of occlusion on gesture recognition was thoroughly analyzed, revealing that the model performs best under no occlusion conditions, with notable declines in accuracy and precision under occlusion and mixed conditions. Additionally, higher frames per second (FPS) were positively correlated with increased precision, indicating that increased temporal resolution improves the model’s ability to correctly recognize gestures. These findings highlight the practical implications of the proposed system for real-time translation and accessibility tools, significantly advancing sign language recognition technology, promoting inclusivity, and fostering greater communication accessibility for NZSL users. Future work will explore different camera angles alongside different lighting conditions to further combat occlusion challenges, ensuring continual improvements in the performance of sign language recognition systems.