Many studies have been conducted on handwritten mathematical expression recognition (HMER) based on encoder-decoder architecture. However, the previous methods fail to predict accurate results due to low-quality images such as blur, complex background and distortion. In addition, ambiguous or subtle symbols caused by different handwriting styles are often recognized incorrectly. In this paper, we propose an efficient method for HMER to deal with the above issues. Specifically, we propose a Dual-branch Refinement Module (DRM) to deal with the challenging disturbances. In terms of ambiguous or subtle symbols, we believe that the combination of local and global information is beneficial to recognizing these symbols. Therefore, we design a Local Feature Enhancement Module (LFEM) to enhance local features, which can cooperate with global information extracted by the following transformer decoder. Extensive experimental results on CROHME and HME100K datasets verify the effectiveness of our method.