The Transaction Processing Performance Council (TPC) is initiating the development of a new benchmark standard to address the emerging demands of Generative AI (GenAI) workloads. Building on the success of TPCx-AI, which established a comprehensive framework for evaluating end-to-end machine learning pipelines, this new benchmark aims to capture the unique computational and data-centric characteristics of GenAI systems. Generative AI models, including large language models (LLMs), diffusion models, and multimodal architectures, introduce novel challenges in data processing, model evaluation, and system scalability. The new benchmark is being designed to reflect real-world, production-scale GenAI scenarios. The benchmark will represent relevant end to end scenarios and full-stack evaluation pipeline, including data ingestion, inference, post-processing, and quality assessment. The TPC consortium is actively engaging with industry stakeholders to define representative workloads, including text summarization, code generation, image synthesis, and multimodal reasoning. The benchmark may incorporate a range of performance metrics critical to GenAI, including throughput, latency, model quality and price performance. Emphasis will also be placed on reproducibility, transparency, and auditability, in line with TPC’s established benchmarking principles. This paper outlines the challenges, design goals, methodologies and early architectural considerations for a new GenAI benchmark.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Next Generative AI Benchmark Development

  • Hamesh Patel,
  • Nirmala Sundararajan,
  • Nicholas Wakou,
  • Paul Cao,
  • David Schmidt

摘要

The Transaction Processing Performance Council (TPC) is initiating the development of a new benchmark standard to address the emerging demands of Generative AI (GenAI) workloads. Building on the success of TPCx-AI, which established a comprehensive framework for evaluating end-to-end machine learning pipelines, this new benchmark aims to capture the unique computational and data-centric characteristics of GenAI systems. Generative AI models, including large language models (LLMs), diffusion models, and multimodal architectures, introduce novel challenges in data processing, model evaluation, and system scalability. The new benchmark is being designed to reflect real-world, production-scale GenAI scenarios. The benchmark will represent relevant end to end scenarios and full-stack evaluation pipeline, including data ingestion, inference, post-processing, and quality assessment. The TPC consortium is actively engaging with industry stakeholders to define representative workloads, including text summarization, code generation, image synthesis, and multimodal reasoning. The benchmark may incorporate a range of performance metrics critical to GenAI, including throughput, latency, model quality and price performance. Emphasis will also be placed on reproducibility, transparency, and auditability, in line with TPC’s established benchmarking principles. This paper outlines the challenges, design goals, methodologies and early architectural considerations for a new GenAI benchmark.