<p>Data heterogeneity is a main challenge in federated learning that impairs the performance of single global model. Observing learning tasks based on deep neural networks, one cutting-edge perspective is to train personalized models by performing shared feature representation learning combined with customized classifiers for each client. However, the averaging global feature representation leads to the inconsistency of the feature space between clients, and merely leveraging local datasets is often insufficient for adequately training the classifier, which both cause poor model generalization especially in low sample scenario. In response, we design a personalized federated learning method with multifaceted feature matching and element-wise classifier fusion, which demonstrates the effectiveness of dual improvement strategy on feature representation and classifier under data heterogeneity setting where data scarcity and label distribution shift. In feature matching, to align each client’s feature space for training better representation, local features are matched with the corresponding global category anchors through an improved feature matching loss, which incorporates a contrastive loss function for angle-level guidance and a regularization term for magnitude-level constraint. In classifier fusion, the desired information from the combined classifier based on bias-variance trade-off is captured at the element level through utilizing a classifier fusion strategy via gradient-based learning for learning more suitable classifier. Extensive experiments on four public datasets under different heterogeneous data settings affirm the effectiveness of our method.</p>