This research investigates the detection of American Sign Language (ASL) using deep convolutional neural networks (CNNs) with a focus on the use of linguistic sub-units for sign language interpretation. It combines three types of components derived from appearance data and 2D/3D tracking data using sign-level classifiers. This study compares the effectiveness of Markov Models with Sequential Pattern Boosting in temporal change encoding and discriminative feature selection. The results show that Sequential Pattern Boosting significantly increases resilience and performance. The study reviews several CNN architectures, including 00, VGG16, LeNet, and ResNet, and proposes customized CNN models for better image categorization. The findings shed information on the potential of deep learning techniques to enhance ASL recognition.