Recommendation systems (RS) have been extensively studied in recent years. This work focuses specifically on rating prediction, with the aim of estimating a product’s rating score from user reviews and ratings. Existing methods focused on textual reviews and user feedback to extract more information to address data sparsity and the cold-start problem. However, outliers significantly affected these methods and struggled to effectively capture the full meaning of textual data. To tackle this challenge, in this paper, we introduce a comprehensive data preprocessing method to handle outlier data before model input. Furthermore, addressing the limitations of previous LDA-based topic modeling approaches, we propose a novel model called PreBERT-Rec that leverages BERT representations with unsupervised clustering techniques (e.g., KMeans, DBSCAN...) as a powerful tool for topic modeling, enabling more effective information extraction. Extensive experiments on Amazon benchmark datasets demonstrate the superiority of our proposed method, achieving up to a 27.63% improvement over the best previous baseline in rating prediction performance.