CCOptimizer: Resource Configuration Optimizer for Model Cache Pool in Cloud
摘要
In cloud-native AI inference services, model loading has become a dominant contributor to end-to-end latency, and its efficiency directly influences system responsiveness, service quality, and user experience. Despite the elasticity of cloud resources, loading performance remains highly sensitive to cache pool allocation, container bandwidth, and cloud disk configurations. Inefficient resource provisioning often leads to prolonged startup time and excessive operation cost, making it difficult to consistently satisfy stringent Service Level Objectives (SLOs) under dynamic and large-scale deployments. To address these challenges, we present CCOptimizer, a cloud-native cache optimizer framework tailored for AI inference services. We introduce the core features and underlying techniques of CCOptimizer, and we demonstrate its effectiveness on a real-world multi-node Kubernetes cluster with representative foundation models.