Groq, a company known for its efficient language processing unit (LPU), has recently introduced lightning-fast queries and other tasks with leading large language models (LLMs) on its website. This new capability allows users to type or speak their queries with voice commands. According to Groq, the speed at which the queries are processed is much faster than what GPU chips from companies like Nvidia can achieve. Groq’s site engine uses Meta’s open source Llama3-8b-8192 LLM by default, but it also offers larger Llama3-70b, Gemma, and Mistral models, with support for other models coming soon.
This development is significant because it showcases the speed and flexibility of LLM chatbots, appealing to both developers and non-developers. CEO Jonathan Ross believes that the usage of LLMs will increase once people realize how easy it is to use them on Groq’s fast engine. The demo demonstrates various tasks that can be easily performed at this speed, such as generating job postings or articles and making real-time changes.
In a demo, Groq’s engine was able to provide instant feedback on the agenda of an upcoming event about generative AI. It suggested clearer categorization, more detailed session descriptions, and better speaker profiles. When asked for suggestions to make the lineup more diverse, it immediately generated a list of speakers and their affiliations in a table format.
Another exercise involved creating a table of speaking sessions for the following week. Groq not only created the tables as requested but also allowed for quick changes, including spelling corrections and adding additional columns. It even offered translation into different languages. Although there were a few minor bugs in making corrections, they were attributed to LLM-level issues rather than processing limitations. This showcases the potential of LLMs operating at such high speeds.
Groq’s efficiency in AI tasks is attributed to its LPU, which is more efficient than GPUs for inference actions. While GPUs are essential for model training, the deployment of AI applications requires efficiency and low latency. Groq has gained attention by offering its services for free to power LLM workloads and has already attracted over 282,000 developers. The company provides a console for developers to build their apps and allows for seamless swapping from OpenAI apps to Groq with simple steps.
In preparation for his talk at VB Transform, Groq’s CEO Jonathan Ross emphasized the company’s focus on the enterprise sector. Large companies are increasingly deploying AI applications and require more efficient processing for their workloads. Groq’s technology uses significantly less power than GPUs, representing a challenge to the GPU-dominated compute landscape. Ross predicts that by next year, over half of the globe’s inference computing will be running on Groq’s chips.
In terms of user experience, Groq’s engine allows both typed and spoken queries. For voice queries, Groq uses the Whisper Large V3 model from OpenAI to translate the user’s voice into text, which is then used as the prompt for the LLM.
Overall, Groq’s advancements in fast and efficient LLM queries provide users with a seamless and powerful experience. With its potential to revolutionize AI tasks and gain widespread adoption, Groq is poised to play a significant role in the future of AI computing.