The OptiSystem simulation tool is used in this article to evaluate a Wavelength Division Multiplexing (WDM) Radio-over-Fiber (RoF) system that has 100 channels and a bandwidth of 2.5 Tbps over fiber. The performance of the system is studied by using Non-Return-to-Zero (NRZ) and Return-to-Zero (RZ) modulation forms at input power levels of − 5 and 5 dBm, respectively. Even though NRZ and RZ formats are compatible with one another, RZ consistently generates superior Bit Error Rate (BER) values over a range of power levels and distances. The BER values of 1.23E−172 and 2.16E−56 for shorter distances (100–150 km) were higher for the RZ format with 5 dBm input power than they were for the NRZ format. With BER values ranging from 1.06E−121 to 1.50E−09, RZ performed much better than NRZ when the distance was increased to between 200 and 300 km. Both reliability and signal integrity are maintained by the RZ format, which is especially beneficial when the data rate is increased, and the transmission distance is increased. An increase in input power of five decibels (dBm) resulted in a reduction in Bit Error Rate (BER) and an improvement in signal quality, which ultimately led to an improvement in system performance. The optimization of input power and the modulation format for high-capacity optical communication systems are the primary focuses of this study. These new technologies, coupled with advanced modulation methods and dynamic dispersion correction algorithms, should be included in the system to solve issues that are associated with high-speed optical networks and to improve the overall performance of the system.