<p>The rise of social media has made it easier to share information in real time. However, it has also made it easier for rumors to spread quickly, particularly during sensitive events, such as elections. Many of these rumors appear in the form of posts that combine text, images, and emotionally charged captions in ways that can mislead people, making them hard to detect with traditional models. While prior multimodal fusion approaches have shown promise, they continue to face persistent challenges, including semantic misalignment across modalities, noisy user-generated content, and the high computational demands of deep fusion architectures. To overcome these limitations, we propose TB-MMRD, a Transformer-based Multi-modal Rumor Detection framework. TB-MMRD consists of three main components. First, a multimodal feature extraction module uses DistilRoBERTa for textual inputs and VGG-19 for images to capture informative representations while reducing computational overhead. Second, a dual-stage fusion architecture introduces agreement-aware gating before and after Linformer-based attention to suppress semantically inconsistent or noisy features at both early and late fusion stages. Third, a lightweight classification head enables fast and reliable rumor classification. We evaluate our model on Twitter, FakeNewsNet (GossipCo and PolitiFact), and a new Ghana-focused dataset (GhElection). Experimental results show that our model consistently outperforms state-of-the-art baselines in terms of accuracy, F1-score, and robustness to noise, validating the effectiveness of alignment-aware filtering and efficient attention in multimodal rumor detection.</p>