<p>Although self-supervised learning is an effective way to pretrain vision transformers for image classification, it usually requires additional datasets derived from the training dataset, which leads to extra computational cost. In this paper, we investigate the simplest pretext task called clean-to-clean (C2C) task for self-supervised learning, which trains the network to be the identity function for the original training images. We show that this task can be still significantly beneficial for the downstream classification task in various aspects: classification performance, object recognition capability, and dataset-efficiency. In order to identify the main components contributing to the effects of the C2C task, we examine each component of the trained models. As a result, we discover that the patch embeddings and attention scores are the components of the models that are distinguishably determined by the task. Accordingly, when a model is initialized in a way that it produces similar patterns of patch embeddings and attention scores to those of the C2C-pretrained model, this C2C-imitating model shows improved classification performance after finetuning even without the pretext task. Furthermore, in order to identify the main parts learning useful weight parameters from the C2C task, we freeze some parts of the C2C-pretrained model during finetuning and compare their performances for the downstream classification task. We find that the model with the frozen stem part achieves the best downstream classification performance, which indicates that the stem part of the model learns useful weight parameters by the C2C task.</p>