Scholarlink

The demand for LLM inference is soaring, driven by widespread adoption of generative AI. Dr. Lisa Su recently projected the data center AI accelerator market to exceed $500B by 2028—with over 60% tied to LLM and AI inference workloads. LLMs power a wide range of applications with their deep understanding and language generation capabilities. As their use grows, accurate benchmarking becomes essential for making infrastructure decisions and optimizing performance for throughput, latency, scalability, and cost. MLPerf's LLM inference has become the industry benchmark for measuring LLM inference performance, especially among hardware vendors. While valuable for peak performance comparisons, this creates a gap for practitioners who need to evaluate diverse, real-world inference scenarios involving varying architectures, software stacks, and deployment models and different LLM models. What’s needed is a flexible, open benchmark that can simulate realistic workloads and provide actionable insights for end users across a range of operational environments. ScaleBench_AI addresses this gap with an open, extensible benchmarking framework built to address this need, enabling scenario-driven evaluation of LLM inference performance across architectures using real-world REST API loads.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

ScaleBench_AI - Flexible LLM Inference Benchmarking Across Architectures and Environments

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Karthik Krishna,
  • Sarthak Dwivedi,
  • Divya Singh

摘要

The demand for LLM inference is soaring, driven by widespread adoption of generative AI. Dr. Lisa Su recently projected the data center AI accelerator market to exceed $500B by 2028—with over 60% tied to LLM and AI inference workloads. LLMs power a wide range of applications with their deep understanding and language generation capabilities. As their use grows, accurate benchmarking becomes essential for making infrastructure decisions and optimizing performance for throughput, latency, scalability, and cost. MLPerf's LLM inference has become the industry benchmark for measuring LLM inference performance, especially among hardware vendors. While valuable for peak performance comparisons, this creates a gap for practitioners who need to evaluate diverse, real-world inference scenarios involving varying architectures, software stacks, and deployment models and different LLM models. What’s needed is a flexible, open benchmark that can simulate realistic workloads and provide actionable insights for end users across a range of operational environments. ScaleBench_AI addresses this gap with an open, extensible benchmarking framework built to address this need, enabling scenario-driven evaluation of LLM inference performance across architectures using real-world REST API loads.