This study explores how the integration of Large Language Models (LLMs) in story-driven video games may impact the social interactions and storytelling with non-playing characters (NPCs). We conducted a case study where participants experienced interacting with LLM-controlled NPCs in the roleplaying game Skyrim. Using the community based game modification Mantella, an alternative to the traditional scripted dialogue trees used for NPC that is substituted with LLM-powered conversations. The case study was conducted in three phases; first with a structured interview, second by an experiment where the participants interacted with LLM-powered NPCs, and finally by an in-depth semi-structured interview concerning the participants’ experiences. The data was thematically analyzed. Our findings indicate that LLM-powered NPCs enhance immersion and agency, as well as offering a sense of recurring novelty due to the unpredictable nature of LLMs. However, implementing a technology which allows for unrestricted dialogue and storytelling in a game not designed for it, poses challenges. For example, nascent technological issues with the integration of the LLM in the game mechanics may have a negative impact on players’ in terms of immersion and the believability of the NPCs. In addition, we examine the potential for LLM-powered NPCs to enrich game worlds in alignment with an appropriate game design. Based on the results, we suggest the suitable approaches for the usage of LLM-powered NPCs in future game development in regards to story-driven games and other applications where narration is key.