Adding LiDAR for an autonomous driving system will complement the weaknesses of a camera-only solution and enhance its robustness. To fully exploit the multimodal advantage, existing works have proposed various multimodal fusion algorithms to effectively combine LiDAR and camera data for scene and object recognition through 3D semantic segmentation. However, most of these methods leverage softmax-based attention modules for intra-modal feature encoding, and early fusion for inter-modal feature learning, leading to excessive computations and therefore higher latency in semantic segmentation. To mitigate this challenge, we propose the Semantic Segmentation (S2) Agent attention module for 3D semantic segmentation in autonomous driving system using LiDAR and camera. Intra-modal encoding is fully explored instead of early fusion using feature concatenation. We adopt a mid fusion strategy to further reduce computations. Experiments using open benchmark datasets nuScenes and Semantic KITTI show comparable or even better mIoUs than state-of-the-art baseline methods while obtaining better latency performance when compared to the most recent MSeg3D algorithm.