Protein binding site prediction is crucial for drug design, but it is challenging due to the small size of the pockets and the complex interactions of the amino acids involved. Many existing methods use a 3D voxel U-Net to extract single-scale samples. However, this approach may overlook the structured or chemical information of the protein and fail to consider the impact of nearby atoms in the pocket. To tackle these issues, we propose a new protein binding site prediction model (DRIVPocket) based on dual-stream rotational invariance and voxel feature fusion. Specifically, DRIVPocket uses a dual-stream framework consisting of a 3D voxel network and an atomic point cloud network to predict the basic pockets and binding atoms, respectively. In addition, we present a novel feature extraction backbone based on dual rotational invariance attention (DRIA), which combines the advantages of shared dual attention information and the point cloud rotational invariance features. This module can extract the detailed spatial and hidden chemical information of the sample. Finally, we achieve a more robust predicted site by integrating the predictions of the above two networks through a semantic fusion module. Extensive experimental results obtained on four benchmarks demonstrate the merits and superiority of DRIVPocket over the existing state-of-the-art approaches. Our code has been released at https://github.com/lv5misaki/DRIVPocket .