Federated learning aims to utilize multiple client domains without sharing data to train a global model, yet generalizing well to unseen domains remains challenging in this paradigm. Recent works indicate that large vision-language models (LVLM) can be used to improve the model’s generalization ability by knowledge distillation. However, due to the unstructured nature of the text features, client models have poor adaptability to distribution shifts. In this paper, we propose Federated Stylistic Feature Dispatcher (FeSFD) to conduct alignment across clients, which improves generalization capacity in privacy-preserving scenarios. Specifically, client domains’ statistical information is shared, facilitating feature-level augmentation with different domain styles to create novel feature distribution. Also, we design a consistency loss that draws augmented predicted results closer to the original results for restraining semantic understanding of the model. Finally, the dual knowledge distillation further excavates the generalization capacity of LVLM. Our model outperforms state-of-the-art FedDG methods through comprehensive experiments on PACS, Office-Home, and Digits-DG benchmarks. What’s more, we achieve 3.96%, 1.53%, and 4.71% improvement on the above three datasets compared to the baseline, which proves the superiority of our model.