This paper investigates optical soliton dynamics in magneto-optic waveguides governed by a coupled nonlinear Schrödinger-type equation with the \(\beta\) -fractional derivative and Kudryashov’s self-phase modulation. Physically, the equation represents the balance between self-phase modulation and chromatic dispersion, which preserves soliton shape during propagation. The introduction of the \(\beta\) -fractional operator provides a generalized modeling framework that enhances the flexibility of the system and allows for a more accurate description of complex wave dynamics beyond classical integer-order formulations. Using the improved modified extended tanh-function method, we derive a wide spectrum of exact solutions, including bright, dark, combo, singular, rational, Jacobi elliptic, exponential, and Weierstrass elliptic solitons. The fractional order \(\beta\) acts as a tuning parameter that significantly influences the soliton width and propagation speed. A modulation instability analysis confirms the robustness of these solutions against perturbations. The results are relevant for optical switching, signal processing, and stable transmission channels in photonic integrated circuits.
