As social media continues to evolve and expand, detecting fake news has become a more complex and pressing challenge. A thorough understanding of the semantic information embedded in rapidly disseminating multi-modal news content is crucial for effective fake news detection. However, the current approaches mainly focus solely on either the multi-modal content of the news or the relational dynamics between the news and its associated comments (i.e., the propagation structure of the news spread). Unlike previous approaches, this work introduces a novel Multi-modal Attentive Graph Opinion Communicating (MAGOC) Network for fake news detection, that simultaneously models both the propagation structure and multi-modal news content. We design a graph opinion communicating approach that aggregates not only the fine-grained content information of news but also the propagation structure information in the news- comment relation graph. To further leverage visual information, we introduce a novel cross-modality attention network that leverages complementary features from images to obtain a more powerful semantic representation of news. Comprehensive experiments conducted on two widely adopted benchmarks, Fakeddit and PHEME, demonstrate that our approach surpasses current leading techniques in detecting fake news.