Unlike traditional object detection tasks, visual grounding aims to locate a specific target within an image based on a given textual description. Current state-of-the-art methods are mainly categorized into two groups: some employ separate visual and textual encoders to extract features independently from the image and the text, and others leverage textual information to guide the visual encoder in extracting semantically relevant visual features. However, the visual features extracted by the former approach are uniquely invariant no matter how the input textual expression varies; the latter approach adjusts visual features solely using complete textual information, making it susceptible to noise from fuzzy or ambiguous text and leading to performance degradation. In this paper, we propose a framework based on Dynamic Gating and Dual Stream Attention (DG-DSA), which relies on the hierarchical architecture of Swin-Transformer to construct a multi-layer visual representation network as a visual backbone. Specifically, low-quality textual cues are dynamically filtered through early cross-modal interactions, and multi-modal embedded representations are constructed. Furthermore, a fine-grained bi-dimensional optimization mechanism across channel and spatial dimensions is proposed to adaptively enhance text-related visual features. Extensive experiments on two widely used mainstream datasets show that our method performs significantly better than existing state-of-the-art methods. Ablation experiments also profoundly demonstrate the effectiveness of our approach.