Aiming to improve credit card fraud detection performance under extremely imbalanced conditions, this paper proposes a credit card fraud detection framework based on a deep reinforcement learning combined with supervised learning method. Unlike approaches that rely solely on transaction features, this framework incorporates fraud probability scores generated by supervised classifier as part of the state representation for deep reinforcement learning. Meanwhile, the deep reinforcement learning framework of Dueling DQN employs a differentiated reward function that penalizes false negatives more heavily than false positives to encourage high recall rates, which is particularly fit for extreme class imbalance scenarios. Experimental evaluations on two benchmark datasets demonstrate that the proposed method significantly outperforms the baseline Dueling DQN model in terms of recall and F1-score. To investigate the impact of different supervised learning strategies, we integrated the Dueling DQN framework with several representative models. Among these methods, the MLP + Dueling DQN combination achieves a 2% improvement in recall, highlighting the effectiveness of using supervised learning outputs to guide deep reinforcement learning for fraud detection. These results validate the robustness, adaptability, and practical value of the proposed method in real-world financial risk scenarios.