Camera localization in prior LiDAR maps has gained more and more attention due to its lightweight and high precision. However, the well-developed RGB camera-based methods show severe performance degradation in high-speed motion and extreme imaging scenarios. To overcome this problem, we propose an efficient event camera-based localization method in prior 3D LiDAR maps, which utilizes the high dynamic range and microsecond resolution of event cameras to improve the localization accuracy in these challenging conditions. Specifically, we first employ an event-based visual odometry (VO) system to reconstruct local sparse 3D points from events. With the pose prediction from VO, we then match 3D points with maps using the nearest neighbor method to get coarse 3D matching points. Finally, the camera poses and 3D matching points are iteratively optimized by minimizing the distance error of the matching points. To verify the feasibility of the proposed approach, experiments are conducted on the public MVSEC and DSEC datasets. The results show that the proposed method accurately estimates the 6-DoF event camera pose under long trajectories and extreme conditions.