Novel researches in 3D object detection have witnessed the emergence of methodologies that fuse LiDAR points with image virtual points through depth completion techniques. However, the inherent density of virtual points from images has been identified as a computational bottleneck. While existing approaches employing fixed binning strategies for scene partitioning and virtual point discarding have demonstrated improved detection accuracy to some extent, they inevitably compromise feature continuity at scene boundaries. Moreover, stochastic virtual point elimination risks the loss of geometrically critical points. To cope with these limitations, a novel framework, GMDF, is proposed for virtual point-based 3D object detection. GMDF implements local density computation directly through 3D gradient manifold mapping, enabling importance-aware sampling that preserves structural integrity. Extensive evaluations on the KITTI dataset show that GMDF obtains state-of-the-art performance, with 3D AP of 92.99% and 85.09% in easy and hard car class, respectively.