Over the past decade, several modern technologies have emerged, providing various services to subscribers. However, this requires a large data rate and high speed of data transfer in mobile communications, which demands an advanced generation to meet users’ needs, which has been increasing recently. Therefore, the next generation 5G should contain all the technical coverage standards in wireless communications. Hence, the concept of using Rof for applications. However, there was a problem of nonlinear effects (srs, sbs, spm.xpm, fwm) which causes the system some failures in the performance of the entire system, especially regarding the transmission distance of (100 to 500 km) as well as the effect on the data rate. For instance, in the fifth generation the downloaded data are from 10 to 100Gbps. Therefore, the researcher will work on data processing using digital signal processing techniques, which are unconventional techniques, where distortions that affect the fifth generation are eliminated using the (Inverse Gaussin) filter. To measure the performance of the system some parameters such as EVM and BER are used. The main idea in signal processing is to use DP-IGA-NG5-RoF in transmission and reception, as the defects occur since the beginning of the signal transmission within the modulation 256QAM, 128QAM, 64QAM until the signal reaches the reception through the power Puget account. At each stage in the proposed system, DWDM technology is used to implement the fifth generation radio-optical network. In that network, the Invers Gaussian filter helps reduce and maintain plus shaping and thus reduce the elastic and inelastic nonlinear deformations that affect the network. Hence, the filtering process resulted in EVM 9.5. over a transmission distance from 100 n to 300 km, and this BER reached E–0.25003677 which is within the acceptable performance values ​​of the proposed project. It is worth knowing that the system was simulated using optisystem 18.