In current Hyperspectral Image Classification (HSI) research, efficiently extracting and embedding complex spectral-spatial information remains a major challenge. On one hand, utilizing complex network models such as LSTM and Transformer often leads to high parameter counts and low computational efficiency. On the other hand, the spatial information of land cover (LC) does not manifest uniformly across all spectral bands. Furthermore, spectral-spatial features are prone to degradation during training. To address these issues, we propose Clover-Net, a diversity-aware feature extraction network. Specifically, we introduce the Spatial-Spectral Balanced Rank-Augmented ConvGroup (SBRA-ConvG-roup) for efficient feature extraction. Additionally, in order to address the other two issues, we propose two key strategies: Stacked Local-Global Cross-Attention (SLGCA) and Rank-Aware KL Regularization (RKL-Reg). Extensive experimental results demonstrate that Clover-Net achieves superior performance compared to state-of-the-art methods, while maintaining the fewest parameters.