Future Sight: Fine-Tuning Language Models for Dynamic Story Generation
摘要
The recent surge in the development of attention mechanisms has made it possible for language models (LMs) to produce text on par with humans. Unfortunately, the autoregressive nature of attention-based LM decoders inhibits them from attending to future information. In this paper, we first define the task of future conditioning: the process of motivating an LM decoder to condition on a future event. We then propose the Future Sight story generation pipeline for fine-tuning an attention-based LM decoder on the future conditioning task. We evaluate Future Sight against three datasets—Writing Prompts, ROCStories, and RecipeNLG. We measure the utility of Future Sight on text inter-diversity and entailment against a fine-tuned GPT-2 baseline. Both our novel automated entailment evaluator and human readers found texts generated by Future Sight to more sensibly lead up to a future event. Additionally, we observe a decrease in inter-diversity for batches of predictions with a fixed future event, demonstrating positive conditioning effects of the future. Our intention for the Future Sight pipeline is as a human-in-the-loop system where authors can propose their own future plot events to guide an LM as it drafts a story.