The high consistency of Invariant Extended Kalman Filter (InEKF) is mainly attributed to its excellent trajectory-independent property. However, when traditional InEKF is applied to multi-source inertial-based navigation that simultaneously include left-invariant and right-invariant observations, the observation matrix would be unavoidably trajectory-dependent, leading to the inconsistency in the covariance update. By introducing covariance switch between the left error and the right error, InEKF can achieve better estimation results owing to more accurate covariance update. The specific approach to incorporate covariance switch into InEKF is proposed in this paper. Moreover, a theoretical analysis of the covariance switch’s effectiveness in enhancing estimation accuracy is conducted. Experiments demonstrate that introducing covariance switch can expedite the convergence comparing with the traditional InEKF, and the attitude errors can be significantly reduced during the transient phase.