The growing popularity of AI inference services has created a substantial demand for AI accelerators. However, the strict tail-latency requirements of inference workloads often conflict with the throughput optimization objectives of these accelerators. Many AI accelerators (e.g., Ascend NPUs, Kunlun chips) support co-located deployment of inference tasks via temporal sharing to improve throughput, where interference between tasks can be abstracted as kernel queuing delays. Motivated by this observation, we design nShare, a system that detects interference at the kernel level on hardware supporting temporal sharing and performs millisecond-scale scheduling control. nShare models interference on temporal shared devices as SLO (Service Level Object) slack and incorporates a dynamic batching mechanism to improve utilization without violating latency constraints. Compared to baseline systems, nShare improves throughput by 39.48%–51.53% while meeting the 99th-percentile SLO.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Millisecond-Level Interference-Aware Scheduling for Multi-Inference Co-Location on Ascend NPUs

  • Wenhao Huang,
  • Fupeng Li,
  • Laiping Zhao,
  • Yeju Zhou,
  • Keqiu Li

摘要

The growing popularity of AI inference services has created a substantial demand for AI accelerators. However, the strict tail-latency requirements of inference workloads often conflict with the throughput optimization objectives of these accelerators. Many AI accelerators (e.g., Ascend NPUs, Kunlun chips) support co-located deployment of inference tasks via temporal sharing to improve throughput, where interference between tasks can be abstracted as kernel queuing delays. Motivated by this observation, we design nShare, a system that detects interference at the kernel level on hardware supporting temporal sharing and performs millisecond-scale scheduling control. nShare models interference on temporal shared devices as SLO (Service Level Object) slack and incorporates a dynamic batching mechanism to improve utilization without violating latency constraints. Compared to baseline systems, nShare improves throughput by 39.48%–51.53% while meeting the 99th-percentile SLO.