Illustration generators have been widely studied to support artistic creation activities. Accurate evaluation of model performance contributes to exploring models for better generative quality. However, conventional evaluation metrics for generative models may provide incorrect results for illustration generators. These metrics are based on a feature extractor trained on natural images and are known to perform poorly on out-of-domain data. In this study, we explore the effectiveness of evaluation metrics for illustrations, which has been disregarded in previous studies. With the intentionally degraded illustration dataset, we found differences in the scores depending on the training method and data of the feature extractors. Furthermore, we demonstrate that metric scores calculated with deep models that can extract representations specific to the illustration domain correlate more with human evaluation results obtained from a subjective experiment. Our findings will provide a basis for evaluating illustration generation studies. Additionally, we created a Python package for evaluating generative models to facilitate studies on deep feature-based metrics, which is available at this URL .