Text Image Machine Translation (TIMT) is a machine translation technology that directly translates source language text images into target language text. The existing end-to-end TIMT model performs well in translation efficiency. However, when processing Vietnamese text images, it fails to fully capture the edge features of characters, resulting in a significant decrease in the translation performance of similar Vietnamese characters under low-quality image conditions. To address this problem, this paper proposes a novel fine-grained contrastive learning module. Specifically, we employ image synthesis techniques to simulate the degradation process of low-quality images in real-world scenarios and design an image-frame-level contrastive unit generation module to enhance the model’s perception of similar character edge features via contrastive learning. Experiments on three self-constructed multi-scenario test sets show that our method achieves significant improvements in Vietnamese-to-English (Vi \(\Rightarrow \) En) and Vietnamese-to-Chinese (Vi \(\Rightarrow \) Zh) translation tasks, with average BLEU score increases of 2.9 and 2.5 points respectively.