Advances in machine learning enable solutions to increasingly complex problems. However, the predominant focus on predictive accuracy in many models often results in insufficient attention to potential biases against certain groups, thereby highlighting the critical need for fairness-aware machine learning. While most existing studies focus solely on debiasing with respect to a single sensitive attribute (e.g., race or gender), they fail to simultaneously consider fairness under multiple sensitive attributes. Furthermore, current fairness-enhancing approaches frequently degrade model performance. To address these limitations, we propose a novel framework named BFPM that achieves a better balance between fairness and performance across multiple sensitive attributes. BFPM consists of two parts. First, in the data pre-processing stage, we generate synthetic samples to balance the proportion of multiple sensitive attributes in the dataset, thereby enhancing fairness. Second, in the in-processing stage, we employ a retrieval-augmented model to obtain the context of each sample, thereby strengthening its representation. Comprehensive experiments across benchmark datasets demonstrate that BFPM significantly outperforms state-of-the-art methods, simultaneously improving fairness while maintaining or enhancing performance.