Driver distraction detection (DDD) holds immense significance in bolstering road safety. Many DDD methods focus on traditional aspects, such as gaze or manual distractions, with less involvement in cognitive distraction. Moreover, the limited research on cognitive distraction also lacks interaction between drivers and driving scenarios. This study aims to bridge this gap by leveraging interaction between driver’s eye movement and DashCam Image (DCI) data to detect cognitive distraction. Specifically, we introduce the Driver Cognitive Distraction Detection (DCDD) model. This model incorporates the driver’s gaze points, captured by an eye tracker, and the DCI as inputs. To capture the spatio-temporal features of the driver’s eye movement, we devise a preprocessing technique that generates an eye movement heat map (Map), which highlights the driver’s areas of focus within the DCI. Then, a Fusion Adversarial Network (FAN) designed, which utilizes the Sigmoid activation function to activate DCI features with Map features. Subsequently, a shared parameter network is employed for adversarial learning, which also focuses on the inverse features activated by Sigmoid activation function. To further achieve the extraction of spatio-temporal information, we propose the Multi-View Multi-Scale Spatio-Temporal Transformer (MMSTFormer). This model excels at capturing spatio-temporal details by integrating multi-view and multi-scale features through cross-attention and self-attention mechanisms. By comprehensively combining spatial and temporal data, our DCDD model achieves remarkable accuracy, attaining a accuracy of 96.45%.