In our current research project we develop a critical framework to facilitate the design and the critique of audio-based interactive narrative experiences. We foreground deep material connections to sound and affect as well as to the rhetorical and narrative properties of story/voice as a vehicle for storytelling. We draw inspiration for our framework from a range of philosophical, theoretical and aesthetic influences including posthumanism, new material feminism, affect studies, interaction design, as well as sound design and interactive and traditional narrative research. Our framework is intended to aid in the design of such audio-based artifacts to support embodied user experiences where affect is a key principle to support innovative material storytelling/deep listening with sound as the primary vehicle for delivery. To facilitate deeper understanding of the framework, we apply it to three original audio-based narratives we created to assess them preliminarily in terms of the four key areas we identify to understand affective narrative audio-based media: Temporality, Mediation, Interaction, and Embodiment/Material Experience. In future research we hope to expand and apply our model to other sound based works and to undergo more intensive user-testing.