Semantic segmentation of 3D point clouds plays a crucial role in tasks such as 3D reconstruction. However, traditional methods often rely on a large amount of manually annotated data for deep neural network training, which is costly and time-consuming. To reduce the need for annotations, weakly supervised semantic segmentation (WSSS) has become a research focus, aiming to train models with minimal labels. However, due to limited annotated data, the network struggles to fully capture the rich geometric structures and semantic details of point clouds, and data sparsity and noise further limit the method's performance. To address these issues, this paper introduces a DDLA-Net network that employs dual local self-attention and multi-scale feature fusion. The geometric neighborhood encoding module (GNEM) encodes the local coordinate features of partially annotated point clouds, enabling the network to focus on the positional information of the point clouds. The Dual Local Attention Module (DLA) calculates the feature similarity between two points and uses multi-scale convolution operations to extract representations of point clouds at different resolutions, effectively mitigating information loss due to point cloud sparsity and significantly enhancing the model's ability to perceive small objects and complex geometric structures. Finally, the DLA module and the Adaptive Feature Aggregation Pooling Module (AFAP) are stacked in series to form a Dual Attention Residual Fusion (DARF) module, which enhances the model's ability to handle complex geometric shapes. In the S3DIS Area5 semantic segmentation experiment, the proposed method achieved results comparable to fully supervised methods with only 0.1% of training labels. Furthermore, when using 0.01% of training labels, the mIoU of the proposed method reached 56.64%, surpassing many existing weakly supervised networks and demonstrating the effectiveness of this approach.