This paper designs disturbance observers for both the position and attitude loops in order to address the problem of disturbances in small-scale unmanned helicopters. In order to overcome the singularity problem introduced by Euler angle representation and the ambiguity problem introduced by quaternion attitude representation, a fixed-time terminal sliding mode controller is developed, using a geometric definition of error. The Lyapunov approach is used to find the greatest stable time. Furthermore, the sliding mode controller’s design successfully avoids singularity issues and lowers chattering. Through simulation, the algorithm’s efficacy was confirmed.