Attention mechanisms, and specifically Transformer models, are dominating the field of image captioning. Current mainstream methods typically use grid image features as input and leverage a multi-head attention-based encoder-decoder framework to contextualize information. However, the encoder’s self-attention mechanism only captures visual relationships between fixed-size grid features, failing to account for varying object scales, which impacts visual feature representation and caption quality. To solve this problem, we propose a novel Scale-aware Multi-head Attention (SMA) to fully exploit multi-scale visual features for image captioning via introducing a scale-extension algorithm to attention. In particular, each head in the multi-head attention integrates distinct high-scale features into the fixed low-scale grid features, enabling the capture of more diverse and richer information. Nevertheless, despite SMA’s excellent performance, the visual-language generation task has also sparked a demand for model explainability. To more accurately explain the proposed model, we introduce a multi-scale visual-language explanation network (MVEN) to accumulate relevancy maps from input to output. The explainability relevancy not only aggregates information in a multi-scale context but also generates more interpretable attention heatmaps in both visual and textual-level.