Generative artificial intelligence technologies enable synthetic audio nearly indistinguishable from human audios, posing significant challenges for authenticity verification. Current audio deepfake detection typically employs pre-trained models for forgery feature extraction. However, the evolving nature of audio synthesis systems requires improved detection capabilities, especially accuracy and generalizability. To address this, we propose Dilated Dual-Branch Classifier, a novel framework tailored for efficient and generalized audio deepfake detection. This framework utilize the pretrained XLS-R to extract rich acoustic representations. XLS-R captures deep acoustic representations,while our dual-branch classifier architecture concludes aggregates sensitive contextual information across feature hierarchies. The multi-dilated convolutional classifier branch can expands receptive fields and capture features at different scales, and sensitive layer classifier branch can select useful contextualized representations.Experimental results demonstrate the state-of-the-art performance, achieving Equal Error Rate(EER) of 1.82% on ASVspoof 2021 Deepfake and 7.14% on In-the-Wild. Source code will be made publicly available upon publication.