<p>To address the issues of significant chattering amplitude and low estimation accuracy in conventional sliding mode observers, a composite nonlinear super-twisting sliding mode observer, integrated with a composite nonlinear extended state observer-based phase-locked loop, is proposed. Firstly, an improved adaptive-gain super-twisting sliding mode observer is proposed to enhance the system’s adaptability. Secondly, a composite nonlinear function is introduced to replace the conventional sign function, and a fuzzy-controlled variable boundary layer is employed to suppress chattering further. Then, the composite nonlinear function is integrated into the extended state observer-based phase-locked loop to form a composite nonlinear extended state observer phase-locked loop, which can extract the rotor position information more accurately compared with the traditional phase-locked loop. Finally, through simulation and experimental verification, compared with traditional schemes, the sensorless control strategy proposed in this paper effectively alleviates the chatter problem caused by discontinuous switching control, improves the estimation performance of rotor position and speed, and enhances the control performance of the system.</p>