Running AI inference comes with various costs, and one of the significant expenses is providing the necessary GPU power. Traditionally, organizations have relied on long-running cloud instances or on-premises hardware to meet their AI inference needs. However, Google Cloud is now introducing a new approach that could revolutionize AI application deployment. With the integration of Nvidia L4 GPUs into Google Cloud Run, organizations can now run serverless inference.
The main advantage of serverless computing is that services only run when needed, and users only pay for what they use. Unlike typical cloud instances that run continuously, serverless services like GPU inference only activate when required. This approach offers efficiency and cost-effectiveness for AI workloads.
Google Cloud Run, a fully managed serverless platform, has gained popularity among developers for its ability to simplify container deployment and management. However, the growing demands of AI workloads, especially those involving real-time processing, have highlighted the need for more powerful computational resources. The addition of GPU support in Cloud Run opens up a wide range of use cases for developers, including real-time inference with lightweight open models, custom fine-tuned generative AI models, and compute-intensive services like image recognition and video transcoding.
One concern often associated with serverless computing is performance. Since the service is not always running, there can be a performance hit during the so-called cold start. To address this concern, Google Cloud has shared impressive metrics for the GPU-enabled Cloud Run instances. Cold start times range from 11 to 35 seconds for various models, demonstrating the platform’s responsiveness. Each Cloud Run instance can be equipped with one Nvidia L4 GPU, providing ample resources for common AI inference tasks.
Google Cloud aims to be model agnostic, allowing users to run any models they want. However, for optimal performance, it is recommended to run models under 13B parameters. As for cost savings, serverless computing promises better hardware utilization, which should translate into lower costs. However, whether it is actually cheaper to provision AI inference as a serverless or long-running server approach depends on the application and expected traffic pattern. Google Cloud will update its pricing calculator to reflect the new GPU prices with Cloud Run, enabling customers to compare their total cost of operations on various platforms.
In conclusion, the integration of Nvidia L4 GPUs into Google Cloud Run brings powerful GPU support to serverless computing, opening up new possibilities for AI inference. With improved performance, scalability, and cost-effectiveness, organizations can leverage this technology to enhance their AI workloads and drive innovation in various industries.