Image captioning mostly employs transformer-based architectures to capture detailed semantic information. However, conventional models often struggle with challenges like the large parameter scale and the loss of visual details, which limit their overall performance. To address these issues, we propose a novel image captioning framework, \(T^2\) Transformer, which integrates a Mean Teacher (MT) learning strategy and a Scale-aware Tuning (ST) module. The MT learning strategy trains a novel transformer-based network, where two distinct language models learn from each other through knowledge distillation and model averaging, leveraging iterative learning to enhance the quality and contextual coherence of generated captions. Meanwhile, the ST module employs a hierarchical convolutional structure, improving the model’s ability to capture multi-scale visual features and facilitating more comprehensive and detailed image content analysis. Extensive experiments on the MSCOCO benchmark demonstrate that our method achieves notable performance while maintaining a small parameter scale.