Permanent magnet actuator (PMA) is a typical electromagnetic actuator used for high-voltage vacuum circuit breaker (VCB). Due to its special properties such as nonlinearity and eddy currents, high-performance tracking control within a timescale of milliseconds has always been a challenging problem. In this paper, a novel velocity tracking control system (VTCS) is successfully designed and implemented. The eddy currents are effectively incorporated into the design of observers and trackers, and a detailed theoretical analysis is provided. In the forward channel, the feedback linearization tracking control from modern control theory is introduced. Benefit by the high-performance nonlinear trackers, it is possible to form a dual-closed-loop control system for velocity and flux linkage by introducing displacement and current as feedback quantities. Through experimentation, the performance of VTCS is analyzed, and its application potential for suppressing bounce of moving contact is verified.