Speech emotion recognition (SER) is crucial in human-comp-uter interaction. In recent years, the transformer has been widely adopted for SER tasks due to its multi-head attention mechanism. However, not all input time frames are important for the emotion classifier, so it is essential to select the time frames rich in emotional information. In this paper, we proposed a Domain Adaptation (DA) network with K-Weight transFormer (KWFormer) for speech emotion recognition, which names DA-KWFormer. The key of KWFormer is the window-based sparse attention mechanism, which selects the top-k most essential features in the temporal dimension, effectively focusing on the emotion-relevant regions of the input features. Furthermore, DA is employed to learn domain-invariant emotion representations to alleviate domain shifts caused by speaker differences in speaker-independent tasks. Experimental results show that DA-KWFormer achieves the best performance with the weighted accuracy (WA) of 74.18%, 96.27%, and unweighted accuracy (UA) of 75.18%, 95.71% on the IEMOCAP and Emo-DB datasets.