<p>Coaxial octorotors have gained widespread application due to their superior control performance and robustness compared to quadrotors. However, the introduction of four more rotors significantly increases the complexity of modeling and control, as the intricate rotor interactions demand a more comprehensive representation of the UAV’s dynamics. Traditional approaches, such as the Newton–Euler method, often separate position and attitude dynamics, leading to redundant operations and increased computational load, which becomes even more problematic for coaxial octorotors. Twistor, as a novel rigid-body representation parameter, is particularly well suited for modeling coaxial octorotors, offering a unified, non-redundant framework that simplifies the description of UAV motion. To address the computational efficiency issues associated with dynamics modeling using the Newton–Euler method and dual quaternions, this study proposes a novel approach based on twistor, which enables a more concise and efficient unified description of UAV pose dynamics. To address the challenges posed by the strong dynamic performance of coaxial octorotors, we design a finite-time double-hyperbolic sliding mode controller for dynamic pose tracking, rigorously proven to ensure global finite-time stability using Lyapunov finite-time criterion and LaSalle invariance principles. The effectiveness and practicality of twistor-based modeling as well as twistor-based controller are demonstrated through hardware-in-the-loop simulations on the RT-Links platform, highlighting its potential for real-world applications in complex environments.</p>