Language knowledge plays a crucial role in understanding the semantic aspects of character sequences. However, in current SOTA end-to-end Scene Text Recognition methods, semantic information is typically treated as an independent module, applied during the post-processing stage of the output sequence. This approach does not fully leverage the semantic information of characters, leading to an inability to provide a more accurate and comprehensive understanding of the visual context. To address this issue, this paper introduces a novel approach, namely Multi-modal Scene Text Spotting Networks: Interactive Enhancements between Visual and Semantic Features (VSENet). First, the visual features and semantic features are concatenated in dimensions, forming a pseudo multi-domain sequence. This sequence is then input into a Transformer-based multi-modal encoder for mutual learning and enhancement. The semantic information effectively enhances visual features, reducing input noise within the visual domain, and increasing confidence in parallel predictions. Moreover, VSENet efficiently extracts features from text of different scales, thereby reducing background interference. An attention mechanism is introduced in the text detection process, and a novel feature extraction structure is designed. Furthermore, extensive experiments, including both English and Chinese, demonstrate significant improvements in accuracy and speed compared to other recognizers when adopting our language modeling approach.