Currently, home electrical appliances are commonly operated by buttons or remote controls, which can be inconvenient due to physical movement or device misplacement. Gesture recognition offers a more intuitive alternative, but existing methods using wearable devices or depth cameras have usability and equipment limitations. While gesture recognition using standard video is less burdensome, it faces challenges such as low accuracy and unintentional gesture recognition. To address these issues, this paper proposes a multimodal recognition system combining video-based gesture recognition and speech recognition. By detecting both fine finger movements and broad upper-body gestures separately and integrating them, the system improves recognition accuracy. The combination with speech input helps prevent misrecognition and allows intuitive control of multiple appliances with common gestures. We implement the proposed system and evaluate the performance.