Recognition of sign language, an essential component to connect the physically impaired community to the rest of the world. Sign language is a well-structured set of gestures for communication. With advancements in computer vision, recognition of sign language becomes possible to others. The proposed study enhances the sign language recognition system by detecting particular gestures. Proposed study detects the individual gesture in the input frame. Proposed study used a hybrid YOLO architecture to enhance gesture detection. Proposer detection of gesture and sequence of gesture helps to improve communication with the impaired community. A system based on gesture detection helps to improve human-computer interaction models. Proposed study used data augmentation to design a model that is more generalized. Simulation of proposed study has achieved mAP50–95 as 0.81. Simulation of the study was also tested on standard YOLO5 and other detection models.