Efficient data communication represents a challenge in wireless data transmission and reception systems, especially in scenarios with intersymbol interference and noise in the communication channel, which degrade the quality of the received information. Two methods are commonly employed to address interference and noise problems: ( \(\varvec{i}\) ) equalization methods to compensate for signal distortions and ( \(\varvec{ii}\) ) error detection methods to identify corrupted data packets. However, these methods have limitations: neural network-based equalization methods categorize all patterns, even in uncertain scenarios, compromising performance, and error detection algorithms are prone to failure, especially in high-noise scenarios. In this paper, we propose a neural network with reject option that simultaneously provides benefits for equalization and error detection. Our approach offers advantages over conventional methods by classifying signals on the basis of their confidence levels. The reject option technique introduced in this paper enhances neural network performance by avoiding classifications with a high risk of error. First, we analyze our proposed neural network using three conventional neural networks for channel equalization. The results indicate that our approach improves performance metrics. After that, we analyze our neural network with a state-of-the-art algorithm by examining bit error rate curves for various communication channels. Finally, we present the results of our method, which are compared with established error detection methods and real data from hardware simulations. Our proposed method can be applied to detect errors without additional data overhead and outperforms other techniques in high-noise scenarios.
