Instance-guided anime editing with a curated large-scale dataset
摘要
Anime content, such as Japanese-style illustrations, manga, and animation, is popular worldwide among diverse audiences. However, editing and repurposing this content for an enhanced viewing experience is complex and relies heavily on manual processes, due to the challenge of automatically identifying individual character instances. Therefore, automated and precise segmentation of these elements is essential to enable various anime editing applications such as visual style editing, motion decomposition and transfer, and depth estimation. Most state-of-the-art segmentation methods are designed for natural photographs and do not capture the intricate aesthetics of anime-style characters, which reduces segmentation quality. The primary challenges are the lack of high-quality anime-dedicated datasets and the absence of competent models for high-resolution instance extraction on anime. To address these issues, we introduce a high-quality dataset of over 100k paired high-resolution anime-style images and their instance labeling masks. We also present an instance-aware image segmentation model that generates accurate, high-resolution segmentation masks for characters in a wide variety of anime-style images. Furthermore, we show that our approach supports segmentation-dependent editing applications such as 3D Ken Burns effects, text-guided style editing, and puppet animation from illustrations and manga.