Contrastive learning for scene text recognition (STR) task greatly relieve the problem of relying on large scale of synthetic data or labeled data for training. Most of previous STR method using contrastive learning attempted to divide the visual features from encoder into fixed number of instances. However, fixing the number of instances may cause the mismatching between positive and negative samples in contrastive learning, especially for data with extreme geometric transformation. To tackle this problem, we introduce a very weakly labeled text length information to restrict the projection heads of the visual feature. Moreover, we found that the attention for the time-step character prediction of the same text in different shapes are different. That inspires us to think about extracting the shape-variation features of scene text, which is equal as making the attention be consistent for the same text with different shapes during decoding stage. In this paper, we propose a Counting-Aware Contrastive Learning Model (CALM) and a plug-in Attention Alignment Module (AAM) to overcome the above challenges for STR. CALM uses the text length information as guidance to determine the projection head numbers for different scene text images in the pre-training of encoder stage. AAM constrains the character attention at each time step regardless of the shape and length of text in the decoder stage. To validate the method, we synthesize the scene text data by sever transformation. Extensive experiments demonstrate the effectiveness of CALM and AAM for STR task. The comparison results also show that our proposed method achieves state-of-the-art performance on public benchmarks.