In autonomous driving, integrating data from complementary sensors is crucial for navigating complex scenarios with dense traffic and dynamic agents. Previous state-of-the-art methods, particularly TransFuser, experience feature loss due to downsampling from pixel-wise cross attention with a transformer encoder, hindering their ability to grasp the global context of traffic situations. To address this, we introduce ViTFuser, a self-attention-based method that utilizes Vision Transformers (ViTs) at multiple resolutions to fuse perspective and bird’s eye view feature maps. The incorporation of a Feature Pyramid Network (FPN) enhances multi-scale feature fusion, improving traffic object detection and enabling global contextual reasoning in the fusion process. We validated ViTFuser on two widely recognized benchmarks, Longest6 and Town05, achieving Driving Scores (DS) of 55.15, 91.07, and 74.95 for Longest6, Town05 short, and Town05 long, respectively, outperforming several previous works in the field.