Modal analysis critically enables engineers to characterize composite shafts’ vibrational dynamics and optimize their mechanical performance by harmonizing fiber orientation, stacking sequence, and stiffness, ensuring robust rotating system design. This paper investigates the modal characteristics of composite shafts, highlighting the impact of design parameters on natural frequencies values. Using Hamilton’s principle, the governing equations of motion for the shaft were formulated, integrating stiffness and damping properties. A finite element model based on the Equivalent Single Layer Theory (ESLT) was implemented to model the composite shaft’s anisotropic properties. This research emphasizes the influence of several factors such as material types, fiber orientation and stacking sequence, on the modal behavior of composite shafts. Various composite materials, including Bore/Epoxy and Carbon/Epoxy, were examined to determine their natural frequencies. Results indicate that optimizing key parameters can significantly enhance the composite shaft’s performance. Specifically, the natural frequencies of the Carbon/Epoxy (C/E) shaft reach their maximum when fibers are aligned at 0°, while adding additional layers increases stiffness, raising natural frequencies further. It is significant that higher material stiffness directly correlates with elevated natural frequencies. These findings collectively demonstrate that strategic parameter selection improves stiffness-to-weight ratios, refines vibrational response, and mitigates resonance risks in rotating systems.