Existing cognitive screening tools for aviators have not focused on domain-relevant functions of cognition, like situation awareness (SA). Without this focus, general aviation (GA) pilots may be vulnerable to undetected cognitive incapacitation, increasing their risk of an accident or mishap in flight. The current research reports on the validation of the CANFLY, an online cognitive health screening tool that indexes cognitive factors critical to safety in GA, such as SA. The CANFLY aims to be a tool that allows pilots to self-assess their pilot-cognition in an “on-demand” fashion. In the present study, pilots completed the CANFLY, which included watching and interacting with pre-recorded flight-segment videos. Pilots monitored radio communications while tracking their flight instruments through adjusting a virtual slider to match a specified instrument gauge. After each video, pilots answered SA questions about their ownship as well as details about other traffic. Pilots then flew a virtual reality (VR) simulated flight scenario, including take off, cross-country flight, and circuit/pattern activities (i.e.: touch-and-gos and landing). The VR flight served to ground true SA abilities of pilots. The VR scenario was paused three times and pilots reported on their ownship and knowledge of traffic, similar to SA tasks contained in the CANFLY. Preliminary results with 27 licensed pilots, ages 19–75 (M = 47.57, SD = 17.61) demonstrate a moderate relationship between SA scores from the CANFLY and the VR flight. These initial findings are important for external validity of the CANFLY, and efforts to make it available for the GA community.