AI models that perform classification and prediction tasks are commonly assessed using established quantitative metrics, but evaluating text-based generative models presents unique challenges. These models generate novel, unpredictable content, often requiring more nuanced evaluation methods that go beyond rigid quantitative scores. In this paper we present a framework that can aid with the evaluation of narrative generation approaches that target screenplays as their output. Screenplays offer formatted metadata and are widely accessible, making them well-suited for automated data analysis and comparison. Our framework provides a diverse set of qualitative and quantitative evaluation metrics, including syntactic complexity, sentiment analysis, part-of-speech distribution, and character presence visualizations, to assess narrative coherence and character dynamics. We also show how our framework can be used to determine differences in human-authored and computationally generated screenplays, and provide an outlook at how this analysis can be used to improve computational approaches.