As human-robot teams become increasingly integrated into predominantly human-centric environments, such as workplaces, hospitals, and even our homes, understanding cognitive fatigue on human teammates is crucial for the robots. This chapter explores the challenge of assessing cognitive fatigue in an offline and online manner through sensory data during Human-Robot Collaboration (HRC). We examine state-of-the-art approaches that leverage signals such as Electrocardiography (ECG), Galvanic Skin Response (GSR), speech-based markers, and others to assess Cognitive Fatigue (CF) states in real-time. The chapter discusses the strengths and limitations of each modality, the available datasets and the state-of-the-art algorithms for offline and online CF assessment, and the implications for adaptive systems that can respond to signs of fatigue. By bridging insights from robotics, neuroscience, and human factors, this chapter provides a comprehensive foundation for designing robust, responsive robot teammates capable of acting when human teammates are cognitively fatigued.