Accurate mass detection in dense breast tissue remains a critical challenge in mammographic screening, as overlapping structures often obscure subtle lesions. We observe that a multi-modal vision-language detection framework with strong pretraining over natural images demonstrates remarkable state-of-the-art (SOTA) performance in lesion detection tasks via transfer learning. However, current SOTA does not take advantage of the entire context available to radiologists. Typical mammogram examinations contain two views of each breast for a given patient, and certain lesions are visible in one view, while hidden by dense breast tissue in another. We propose a novel Multi-view Multi-modal DETR object detection framework that trains a network to detect lesions after considering both views, thus emulating the workflow of a radiologist. Specifically, our method incorporates a bidirectional cross-attention fusion module to integrate relevant information from craniocaudal (CC) and mediolateral oblique (MLO) views simultaneously, reinforcing lesion-specific signals and aiding detection of masses that may be obscured by dense tissue. We evaluate our proposal on the public VinDR-Mammo dataset, achieving significant improvements in mass detection with reduced false negatives. Our method reaches a mass detection mAP of 0.654, outperforming Mammo-CLIP (0.580) by an absolute margin of 12.8%. It also reduces the false negative rate in DENSITY C cases by 5.9% compared to the single-view baseline, highlighting the clinical value of our method. The code is available at https://github.com/MMDETR/MM-DETR .