The integration of technological resources in critical healthcare tasks, particularly in surgical settings, necessitates the reduction of contamination risks associated with traditional input devices such as keyboards and mice. Hand gesture recognition systems present a promising solution to this challenge. This chapter introduces a gesture recognition method applied to a medical assistant robot, specifically designed to facilitate instrument delivery and collaboration with surgeons during procedures. The accurate and swift recognition of the surgeon’s hand gestures is imperative for effective instrument passing. In addressing the existing challenges in gesture recognition, data were collected using UltraLeap, and classification was implemented using an ensemble model comprising DenseNet 201, SE-Inception-V3, and SE-SqueezeNet. This accuracy and recognition speed was realized for the purpose of enhancing the interaction between the surgeon and the medical assistant robot. Our approach achieves 95% and 92.2% accuracy on our self-collected gesture dataset and the publicly available Jester dataset, respectively, achieving great improvement in gesture recognition performance.