<p>The integration of affective computing with virtual reality (VR) and augmented reality (AR) has reshaped immersive experiences by enabling emotion-aware systems that can dynamically adapt to users’ affective states. By leveraging physiological and behavioral data, these systems enhance interaction quality, engagement, and personalization across a wide range of immersive applications. However, their deployment introduces critical challenges related to privacy, algorithmic bias, real-time affect recognition, and ethical governance. This survey offers a focused and systematic analysis of the technical, ethical, and social implications of affective VR/AR systems. It examines privacy-preserving techniques (PPTs), cross-cultural emotion recognition, and artificial intelligence (AI)-driven affective adaptation, as well as the risks associated with emotional manipulation and regulatory oversight. Advances in federated learning (FL), reinforcement learning (RL), and neuromorphic computing are reviewed as potential enablers of secure, efficient, and adaptive affective processing in immersive environments. The survey argues that future affective VR/AR systems must balance emotional intelligence with user autonomy through transparency, inclusivity, and ethical responsibilities. By synthesizing ethical theory, technical architectures, and societal considerations within a unified framework, this work addresses a critical gap in the literature and provides guidance for the responsible design and governance of emotion-aware immersive systems.</p>