Genomic DNA encodes diverse regulatory mechanisms that control gene expression across different functional and cellular contexts. Existing genomic foundation models often struggle to generalize across tasks because they rely on shared dense parameters to represent highly heterogeneous sequence patterns. To address this limitation, we propose SparseDNA, a hybrid genomic language model that integrates sparse Mixture-of-Experts (MoE) modules into a state-space architecture. By replacing dense projections with sparsely activated expert subnetworks, our model enables adaptive specialization for fine-grained modeling of heterogeneous genomic signals–such as promoters, enhancers, and histone marks–while maintaining computational efficiency. This design significantly increases the effective parameter count and enhances the model’s ability to capture diverse genomic dependencies without substantially raising training costs. Pretrained on the human reference genome at single-nucleotide resolution, SparseDNA achieves state-of-the-art performance on 10 of 18 datasets from the Nucleotide Transformer benchmark, with an average accuracy gain of 1.3% and a notable 2.0% improvement on histone-related classification tasks. Further, the ablation and controlled comparison studies collectively demonstrate that the performance gain is driven by architectural innovation in expert specialization. These results indicate that combining state-space modeling with conditional expert computation provides a scalable and biologically grounded foundation for modeling the regulatory complexity of genomic sequences.