The complex two-dimensional structure makes the recognition of handwritten mathematical formulas a challenging task, and offline methods cannot utilize stroke order information and trajectory information, further increasing the difficulty of accurate recognition. Traditional scene text recognition methods fail to fully exploit stable and effective feature representations of formula texts. To address this issue, this paper proposes a novel offline handwritten mathematical formula recognition method aimed at utilizing the intrinsic representation of formula images. First, elements in the feature map are modeled as nodes in an undirected graph, and pooling aggregators and weighted aggregators are used to learn the primitive representation, generating a two-dimensional feature map. Subsequently, a Graph Convolutional Network (GCN) is employed to obtain text representation, which is then input into a self-attention network along with positional encoding. By adopting the self-attention mechanism of Transformer, the model can capture long-range dependencies on the two-dimensional feature map, enabling the recognition of arbitrarily arranged and widely spaced characters. A network for learning primitive representations is constructed, utilizing visualized text representations for parallel decoding. The proposed model has been compared with other state-of-the-art (SOTA) HMER techniques on the CROHME 2014/2016/2019 datasets and achieved the best results.