Federated Learning is a distributed machine learning paradigm that enables model training without sharing clients’ raw data. This approach enhances user privacy but incurs large communication overhead due to the extensive number of messages exchanged between clients and the server. Quantization emerges as a prominent solution to mitigate this issue by representing model parameters with fewer bits. However, conventional strategies typically implement quantization at the client level with fixed and unified internal quantization, disregarding the varying significance of different model parameters. In this paper, we explore quantization at both inter-client and intra-client levels, to propose the FedMQ, an innovative framework that implements a balance between communication efficiency and model accuracy. Initially, we devise a multi-grained quantization strategy that determines the quantization bit-depth for model parameters, considering both their importance and the clients’ communication conditions. Subsequently, we design a phased aggregation tactic that enhances model convergence by modulating the influence of quantization errors on aggregated weights in a phased manner. Moreover, we introduce an accuracy compensation technique that periodically adjusts the learning rate to further counteract the accuracy degradation attributable to quantization. Comprehensive evaluations, conducted across a variety of models and datasets, substantiate the advantages of FedMQ. Evaluation results reveal that FedMQ achieves a reduction in communication time ranging from 19.35% to 87.65% while maintaining model accuracy compared to existing baselines.