MLCommons, a vendor-neutral organization that manages MLPerf benchmarks, has released its latest set of MLPerf inference results. These benchmarks offer valuable insights into the rapidly evolving AI hardware and software landscape, with 964 performance results submitted by 22 organizations. The results serve as a vital resource for enterprise decision-makers looking to navigate the complex world of AI deployment. MLPerf provides standardized, reproducible measurements of AI inference capabilities, enabling businesses to make informed choices about their AI infrastructure investments.
One notable addition to MLPerf Inference v4.1 is the evaluation of a Mixture of Experts (MoE), specifically the Mixtral 8x7B model. The MoE benchmark addresses the challenges posed by increasingly large language models. Instead of one large, monolithic model, the MoE approach consists of several smaller models, each specializing in different domains. This approach allows for more efficient deployment and task specialization, potentially offering enterprises more flexible and cost-effective AI solutions.
The MoE benchmark tests performance on different hardware using the Mixtral 8x7B model, which combines question-answering, math reasoning, and coding tasks. The goal is to showcase the capabilities of this emerging architectural trend in large language models and generative AI. By routing queries through different experts, the MoE approach enhances efficiency and performance.
Another exciting development is the debut of Nvidia’s next-generation Blackwell GPU processor. Although it will be some time before Blackwell is available to real users, the MLPerf Inference 4.1 results provide a promising preview of its power. The Blackwell GPU shows a 4x improvement in performance compared to Nvidia’s previous generation product on a per GPU basis. This level of performance gain is impressive and highlights the continuous advancements in AI hardware.
In addition to the Blackwell GPU, Nvidia’s existing GPU architectures, such as the Hopper GPU, continue to deliver impressive performance gains. The MLPerf Inference 4.1 results for the Hopper GPU show up to a 27% increase in performance compared to the previous round of results. These gains are achieved through ongoing software tuning, demonstrating the importance of software optimization in maximizing the performance of AI hardware.
Overall, the latest MLPerf inference results provide valuable insights into the capabilities of AI hardware and software. The addition of the MoE benchmark and the promising performance of Nvidia’s Blackwell GPU and Hopper GPU highlight the continuous advancements in AI technology. For businesses looking to invest in AI infrastructure, these results offer guidance and help in making informed decisions. MLPerf serves as a benchmarking standard that ensures fair and accurate comparisons and enables enterprises to optimize their AI deployments for performance, efficiency, and cost.