As the core perception component, 3D object detection plays a crucial role in autonomous driving and robot navigation systems. However, most existing point-voxel based methods rely on feature conversion and complex fusion or feature aggregation modules based on proposals, which require significant computational overhead. In this paper, we propose an attention based sparse point-voxel network (ASPVNet), which is an efficient point-voxel based approach that extracts features consecutively using point and voxel representations and leverages their respective advantages. Specifically, an attention based sparse point-voxel convolution is introduced that extracts point features using channel attention and the resulting features are used to update the corresponding voxel features, followed by sparse convolution for extracting voxel features. We also propose an auxiliary segmentation module based on 3D heatmap that generates 3D heatmap soft labels to emphasize the foreground points and the overlooked background points. By incorporating category, geometry, and structural information into soft labels can also enhance the network’s ability to perceive structures. In addition, we present a data augmentation method to ease the point cloud density imbalance problem, which uses nearby instances to simulate distant sparse instances to improve the detection accuracy of distant instances. Experiments on the KITTI dataset demonstrate that our method plays favorably against existing state-of-the-art methods.