This study investigates the role of feature extraction and dimensionality reduction in enhancing sentiment analysis performance. We explore the effectiveness of different feature extraction techniques, such as Bag of Words and Term Frequency-Inverse Document Frequency, and apply a simple statistical feature selection method, the Chi-Square test, to improve model efficiency. The models evaluated in this study include Naive Bayes and Logistic Regression, both of which are white-box models that offer interpretability and computational efficiency. Using the IMDB movie reviews dataset, we found that by carefully selecting features and utilizing higher-order n-grams, our models achieved accuracy levels comparable to those obtained by more complex models, such as BERT. Additionally, the simplicity of the models enabled faster training and inference times, processing the data in just a few seconds compared to the several hours required by deep learning models. The results demonstrate that traditional machine learning approaches, when combined with effective feature extraction and dimensionality reduction techniques, can yield high-performance results while maintaining transparency and efficiency.