Surgical instrument segmentation in videos is essential for computer-assisted interventions, enabling accurate tool identification during surgeries. However, current Grounded Conversation Generation (GCG) methods struggle with specifying the instrument of interest and capturing complex interactions in dynamic environments due to limitations in understanding intra-frame and inter-frame information. Here, we formulate a novel Video-GCG framework for improved reference surgical instrument segmentation, which combines visual data with context-aware textual descriptions. First, we develop a Temporal Dynamic Sampling (TDDS) strategy to enhance temporal-spatial feature extraction, solving the intra-frame problem. Then, we present a mask decoding strategy to refine segmentation outputs and reduce the impact of blurred or ambiguous visual information from the surrounding environment, tackling the inter-frame problem. Experimental results show that our method outperforms the state-of-the-art VIS-Net by 18.1% and 7.5% in mAP on the EndoVis-RS17 &18 datasets, showcasing superior performance and efficiency with fewer computational resources. Codes will be released.