Visible-infrared (VIS-IR) image pairs offer complementary information and hold significant potential for UAV-based object detection. However, most existing multi-modal detection approaches overlook the prevalent issue of weak spatial misalignment in real-world scenarios. Although several methods attempt to address this challenge, they often rely on complex architectures for explicit alignment. In this work, we propose DMDet, a novel detection framework based on deformable attention to achieve adaptive and implicit alignment between modalities. To guide effective feature fusion, we introduce a Competitive Modal Feature Selection (CMFS) module that dynamically identifies the most informative modality for each object instance. Furthermore, we design a Deformable Cross-Feature Attention (DeformCFA) module that adaptively samples and aggregates complementary VIS-IR features, effectively mitigating misalignment and enhancing fusion quality. Extensive experiments on the DroneVehicle dataset demonstrate that DMDet significantly outperforms state-of-the-art multi-modal object detectors, especially under weak misalignment conditions.