Driving is a key component of independence and quality of life for older adults. However, cognitive decline associated with conditions such as mild cognitive impairment (MCI) and dementia can compromise driving safety and often lead to premature driving cessation. Conditionally automated vehicles, which require drivers to take over control when automation reaches its operational limits, offer a potential assistive solution. However, their effectiveness depends on the driver’s ability to respond to takeover requests (TORs) in a timely and appropriate manner. Understanding emotional responses during TORs can provide insight into drivers’ engagement, stress levels, and readiness to resume control, particularly in cognitively vulnerable populations. This study investigated affective responses, measured via facial expression analysis of valence (emotional tone) and arousal (emotional intensity), during TORs among cognitively healthy older adults and those with cognitive impairment. Facial affect data were analyzed across different road geometries (straight vs. curved) and speeds (50 km/h vs. 100 km/h) to evaluate within- and between-group differences in affective states. Within-group comparisons using the Wilcoxon signed-rank test revealed significant changes in valence and arousal during TORs for both groups. Cognitively healthy individuals showed adaptive increases in arousal under higher-demand conditions, while those with cognitive impairment exhibited reduced arousal and more positive valence in several scenarios. Between-group comparisons using the Mann-Whitney U test indicated that cognitively impaired individuals displayed lower arousal and higher valence than controls across different TOR conditions. These patterns may suggest blunted emotional activation and potentially diminished situational awareness in cognitively impaired drivers. These results also highlight the potential need for adaptive, conditionally automated vehicle systems that can detect affective states and deliver supportive handover strategies tailored to cognitively vulnerable populations.