Ultrasound imaging for breast cancer diagnosis suffers from reduced segmentation performance due to speckle noise and low contrast, particularly causing high uncertainty at object boundaries, which makes accurate segmentation challenging. To address this issue, we propose the Uncertainty-Guided Graph Neural Network with Frequency Fusion Network (UGFNet), which integrates the Uncertainty-Aware Graph Module (UAG) and Uncertainty-Based Frequency Feature Fusion Module (UFF) into an Attention U-Net framework to quantify and effectively utilize uncertainty in segmentation. UAG employs a Graph Neural Network to distinguish high-uncertainty regions (main nodes) from low-uncertainty regions (sub nodes) and optimize information propagation, while the Main-Sub Uncertainty Loss (MSL) helps facilitate reliable feature learning. Additionally, UFF leverages high-frequency components to recover fine details lost due to ultrasound artifacts and adaptively fuses spatial and frequency-based features to enhance segmentation performance. Experimental results demonstrate that UGFNet outperforms state-of-the-art models on the BUSI and UDIAT datasets, achieving superior accuracy.