Bidirectional Recurrent Neural Networks (BRNNs) are an evolution of the well-known Recurrent Neural Networks (RNNs). However, BRNN’s high accuracy comes at the cost of many training parameters and very complex data and control dependencies compared to Unidirectional RNNs. We propose a novel architecture, Semi-Bidirectional Recurrent Neural Network (SB-RNN), that drastically improves the accuracy of unidirectional RNNs by adding some backward connections between cells of different layers. We evaluate our model considering the PTB data-set, SST and synthetic data using the state-of-the-art deep learning frameworks TensorFlow-Keras. SB-RNN achieved a validation accuracy of 99.38%, outperforming unidirectional and bidirectional models while utilizing fewer resources. Experimental results indicate that SB-RNN matches and often surpasses the accuracy of bidirectional RNN models while they feature a much simpler architecture that requires the same number of parameters as unidirectional designs.