Open-ended video question answering have achieved remarkable progress with the advancement of large language models (LLMs) and vision models. However, most existing methods rely on uniform video frame sampling strategies, which suffer from high computational costs and noise at high sampling rates, and risk missing critical frames at low sampling rates. In this study, we propose Location-Before-Answering (LBA), a multi-scale video segment sampling approach, comprising three modules: a multi-scale video segment generator, a question-related sampling strategy, and a video question answering decoder framework. LBA is designed to precisely identify video segments relevant to a given question while minimizing redundancy and mitigating interference from irrelevant frames. Extensive experiments and analyses conducted on the MSRVTT-QA, MSVD-QA, and ActivityNet-QA datasets demonstrate the effectiveness and superiority of our proposed approach.