Modern embedded systems are characterized by a steady progression towards greater efficiency, complexity, and autonomy. A key aspect of this evolution is the incorporation of self-* properties, enabling embedded systems that adapt and optimize performance autonomously. While self-* properties are fundamental for building autonomous embedded systems, they alone are not sufficient to realize autarkic embedded systems. A truly autarkic system should be able to operate self-sufficient by harvesting energy from its environment, eliminating the need for external power sources or frequent battery replacements, which is of particular interest in the Internet of Things. However, the possibility of unexpected power outages and the consequent loss of data and system states necessitates a fundamental re-evaluation of how modern embedded computer architectures are designed. This article emphasizes the role of the memory system in this context. Here, non-volatile memory (NVM) technologies can retain data even in time intervals of power shutdown or intermitting power outages. In this paper, we present the fundamentals of intermittent computing with unreliable power supply as well as the benefits, design considerations, and challenges associated with integrating NVM into the memory hierarchy of self-powered systems. Finally, we discuss future research directions for enhancing the resilience and efficiency of using NVM in the context of self-powered IoT systems.