Large language models (LLMs) employed in text generation often face bottlenecks due to the sequential nature of text generation. Speculative decoding addresses this issue by using a draft model to propose candidate tokens, which a target model then evaluates. However, it still struggles with inherent delays in token-by-token generation. This paper introduces ‘Masker,’ a method that efficiently identifies and replicates entire segments from the input in one decoding step during long-text QA scenarios by utilizing dynamic masks. Masker leverages overlaps in input-output pairs to expedite text generation, drastically reducing the number of decoding steps required when direct replication from the input is feasible. By precisely predicting replicable segments, Masker allows for complete segment replication in a single step. Built on the theoretical framework of speculative decoding, which ensures the fidelity of the output distribution, Masker delivers competitive speed improvements of up to 2.34 \(\times \) in Long-Text QA scenarios while maintaining accuracy. This approach highlights Masker’s capability to significantly reduce computational demands in scenarios involving extensive queries and answers, demonstrating its suitability for complex QA applications.