Image clustering is a very useful but difficult task in machine learning and computer vision. Recently, convolutional neural network (CNN) is taking a major role in image clustering in which it is used to extract features to be used for an objective function of similarity or dissimilarity measures. There are two problems in the image clustering methods using CNN. First, it is still challenging to estimate a correct number of clusters initially, which decides the quality of the clustering result mostly. Second, a clustering algorithm frequently clusters some images with semantically different contents, into a same cluster (due to their similar color distributions) and generates incorrect clustering result. In this paper we propose a framework for image clustering with CNN and learning from confusion, which can address these problems. The experiments show that the proposed framework increases image clustering effectiveness significantly.