The transformer model offers substantial advantages across various applications, including natural language processing, machine translation, and image captioning. It effectively handles long-range dependencies and allows for parallel computations. However, transformers face limitations in capturing fine-grained dependencies and ensuring efficient information flow between the encoder and decoder. To address these gaps, we optimize the transformer architecture by introducing a criss-cross attention module. This module gathers contextual information in both horizontal and vertical directions. As a result, the model captures intricate dependencies and spatial relationships more effectively, leading to a comprehensive understanding of the input data. In addition, we utilize mesh connections during the decoder stage to leverage the low and high level features. This fully connected approach between the encoder and the decoder facilitates more effective bidirectional information transfer, enhancing the contextual understanding and coherence of the generated outputs. Extensive experiments on the MSCOCO benchmark reveal that our enhanced model generates more accurate and descriptive captions. Our model demonstrates superior performance in metrics such as BLEU, METEOR, and CIDEr.