In the realm of high-speed data transmission, the increasing demand for bandwidth and low-latency communication has intensified the utilization of busy fiber optic lines. These high-capacity channels are pivotal for supporting the vast data flows required by contemporary digital applications, including cloud computing, real-time streaming, and large-scale data analytics. However, the integrity and efficiency of data transmission over these fiber lines are frequently challenged by signal degradation in the form of nonlinear phase noise, frequency offset, four wave mixing and many more. Here, we propose a modified third order Volterra Nonlinear Equalizer that holds a promising practical solution for addressing nonlinear impairments in next-generation optical communication networks, paving the way for improved data transmission reliability and efficiency. In this work, we used 100-Gbaud truncated probabilistic shaped (TPS) polarization multiplexed 64-QAM signal. This manuscript proposes a novel approach for mitigating these effects utilizing a modified third order Volterra Equalizer. By incorporating this function into the system design, we demonstrate enhanced compensation capabilities for SPM-induced distortions, laser source noise, and frequency offset, thereby improving the overall performance in 800 Gbps Polarization Multiplexed 64-QAM system. Its performance is compared with Extended Kalman Filter. Through extensive simulations, we illustrate the effectiveness of the proposed method in achieving superior signal fidelity and reduced error rates in the form of constellation diagram, mean square error and quality factor in optical systems operating at high data rates.