MiDedup: A Restore-Friendly Deduplication Method on Docker Image Storage Systems
摘要
Docker image storage systems, like Docker registry, often employ deduplication to reduce storage overhead. Existing deduplication methods for these systems detect redundancy at either layer or file level, each with its own advantages and limitations. Layer-level deduplication offers much faster restoration but lower deduplication ratio, as it eliminates duplicate layers while ignoring duplicate files across layers. Conversely, file-level deduplication offers a higher deduplication ratio but slower restoration, as removing duplicate files scatters the files of a layer, leading to multiple disk read I/O during restoration. Our analysis reveals that the slow restoration in file-level deduplication partly stems from ignoring file-layer relationships. In this paper, we propose MiDedup, a new Docker deduplication method. MiDedup detects redundancy at file-level and organizes files into a restore-friendly layout by considering file-layer relationships to enhance restore speed. Our studies also show that duplicated files of one layer are not uniformly distributed across different layers. MiDedup uses this insight to skip deduplicating scattered files to further optimize the file layout for faster restoration. Evaluations indicate that MiDedup achieves restore speeds that are 8–19 \(\times \) faster than file-level deduplication (slightly slower than layer-level one), while maintaining 9–47% higher deduplication ratios than layer-level deduplication.