A emerging trend in addressing fatigue driving is real-time monitoring of driver’s fatigue state. In this paper, a driver fatigue detection scheme based on multiple behavioral characteristics and adjacent state information is designed. Linear weighting is applied to behavioral characteristics of driver’s eyes, mouth and head to obtain state parameter. Then, the state parameter is fused with adjacent state information to get state judgment indicator that represents wakefulness, mild fatigue, severe fatigue, etc. The state judgment indicator and two sets of dual thresholds are used to achieve driver state detection. The innovation of the scheme is reflected in two aspects: the fusion of state parameter and adjacent state information and the introduction of two sets of dual thresholds. Simulation results show that compared with the scheme based on state parameter and single threshold, the scheme designed in this paper improves the accuracy by about 10%.