Deep learning models achieve remarkable performance on balanced datasets but struggle with real-world long-tailed data due to dual imbalance: in sample quantity and learning difficulty. Most existing methods focus primarily on re-balancing class sample quantity, neglecting the critical dimension of difficulty imbalance among classes, which limits further optimization. In this paper, we argue that effectively addressing long-tailed recognition requires handling this dual imbalance and propose a novel Gradient Distribution Balancing (GDB) method. The key insight is to dynamically maintain a fine-grained three-dimensional gradient distribution matrix during training to evaluate the learning effect of each category and the mutual influence among them. Leveraging this, GDB intelligently improves the loss function with a dual-balancing mechanism. It reduces the negative gradient suppression from head to tail classes and simultaneously adjusts weights for hard classes to address the quantity and difficulty imbalance respectively. Experiments on long-tailed CIFAR-10/100-LT and MSCOCO-GLT benchmarks demonstrate that GDB significantly enhances tail-class recognition without compromising head-class performance, proving a more balanced and effective solution.