Magnetic encoders are widely used in aerospace and precision manufacturing for high-precision control of motor position and rotational speed, serving as a crucial source of angular feedback. However, magnetic encoders are susceptible to nonlinear measurement errors under non-ideal conditions, such as installation errors. This study systematically analyzes the sources of these errors and designs a measurement apparatus to capture the error curve for the angular acquisition of a miniature control moment gyroscope (mCMG) frame. The whale optimization algorithm (WOA) is introduced to optimize key hyperparameters of the RF model, resulting in a 13.3% reduction in root mean square error (RMSE) compared to manual tuning. Finally, the trained error model is applied for error compensation in the magnetic encoder, significantly improving its accuracy from 0.29127 to 0.01913. These results validate the effectiveness of the proposed error compensation method.