Effective intrusion detection in modern networks is often challenged by two key issues: the proliferation of sophisticated attack variants and the difficulty in modeling complex feature interdependencies within tabular network traffic data. To address these challenges, we propose XGTFormer, a novel hybrid framework that integrates the complementary strengths of Transformer-based feature learning and gradient-boosted decision trees. The architecture leverages a dedicated feature tokenization mechanism that unifies heterogeneous numerical and categorical attributes into cohesive embedding sequences. These sequences are processed through multi-layer Transformer encoders, where multi-head self-attention dynamically captures high-order feature correlations and long-range dependencies. The position-wise feedforward layers then refine these representations into discriminative patterns. To further enhance robustness, we adopt a dual-strategy approach: (1) A probability-level fusion technique that combines the classification outputs from Feature Tokenizer Transformer (FTT) and XGBoost using weighted averaging, capitalizing on FTT’s ability to extract complex patterns and XGBoost’s efficiency in decision-making with rule-based boundaries; (2) An integrated cost-sensitive learning scheme that addresses class imbalance by applying inverse class-frequency weighting to the cross-entropy loss function, boosting model sensitivity to minority attack classes. Comprehensive evaluations demonstrate XGTFormer’s superior detection capabilities. On the CICIDS2017 dataset, it achieves remarkable results (Accuracy: 0.9974, F1-Score: 0.9779), and on the more challenging UNSW-NB15 dataset, it maintains strong performance (Accuracy: 0.8584, F1-Score: 0.8903). These findings validate XGTFormer as an effective solution for high-performance intrusion detection in tabular network data, addressing the complexities of feature interactions and class imbalance.