In autonomous driving, visual features play a crucial role. End-to-end driving models (E2EDMs) extract numerous visual features from the driving environment to solve driving tasks. However, these visual features are often difficult for humans to understand, leading to explainability issues. This study aims to improve the explainability of E2EDMs by enhancing their ability to extract semantically meaningful and driving-related visual features, like vehicles, pedestrians, and traffic signals. The training process of E2EDMs involves leveraging a backbone that is pre-trained on large datasets and subsequently fine-tuned for driving tasks. To address the explainability issue of E2EDMs, previous studies have designed complex E2EDMs during the fine-tuning stage. In this paper, we enhance the explainability by improving the backbone’s ability to recognize driving-related features, i.e., object features. We propose CROp-based COntrastive DIscriminative LEarning (CROCODILE), an additional pre-training method for the backbone. CROCODILE improves the backbone’s ability to preserve driving-related features while suppressing irrelevant features. Then, during fine-tuning, only driving-related features will be used for driving action prediction, thereby achieving high explainability. In addition, CROCODILE eliminates the need for complex structures in the fine-tuning stage.