Anthropic, an AI company, has recently introduced prompt caching on its API, revolutionizing the way developers interact with the models. Prompt caching allows users to store frequently used contexts, enabling them to add additional background information without incurring additional costs. This feature has been made available in public beta on Claude 3.5 Sonnet and Claude 3 Haiku models, with support for the largest model, Opus, coming soon.
Prompt caching has garnered significant attention and positive feedback from early users. They have reported substantial speed and cost improvements across various use cases. These use cases range from including a full knowledge base or 100-shot examples to embedding entire documents in a prompt or incorporating each turn of a conversation. By leveraging prompt caching, developers and users can fine-tune model responses, leading to more accurate and tailored outputs.
One of the major advantages of prompt caching is its impact on pricing. Anthropic claims that using cached prompts significantly reduces costs compared to the base input token price. For instance, the cost of writing a prompt to be cached on the Claude 3.5 Sonnet model is $3.75 per 1 million tokens (MTok), while using a cached prompt only costs $0.30 per MTok. This upfront investment in caching prompts can result in a 10x savings increase for subsequent API calls.
While the initial API call is slightly more expensive to account for storing the prompt in the cache, all subsequent calls are priced at one-tenth of the normal rate. This pricing model encourages developers to utilize prompt caching, as the long-term cost benefits outweigh the initial investment. Users of the Claude 3 Haiku model can expect to pay $0.30/MTok to cache prompts and $0.03/MTok when using stored prompts.
Although prompt caching is not yet available for the Claude 3 Opus model, Anthropic has already announced the pricing for this feature. Writing to cache on the Opus model will cost $18.75/MTok, while accessing the cached prompt will cost $1.50/MTok. It is important to note that Anthropic’s cache has a 5-minute lifetime and is refreshed with each use.
Prompt caching is not a unique concept, as other AI platforms also offer similar features. Lamina, an LLM inference system, utilizes KV caching to reduce GPU costs. OpenAI, a prominent player in the AI industry, has also received inquiries about caching prompts on their developer forums and GitHub. However, it is worth noting that prompt caching differs from large language model memory. While models like OpenAI’s GPT-4o offer memory for storing preferences or details, they do not store actual prompts and responses like prompt caching does.
In the highly competitive AI landscape, Anthropic has been striving to differentiate itself through pricing strategies. Prior to the release of the Claude 3 models, Anthropic reduced the prices of its tokens, aiming to attract third-party developers by offering cost-effective options. This move puts Anthropic in direct competition with industry giants like Google and OpenAI, who also offer low-priced alternatives for developers.
In conclusion, prompt caching is a game-changer for developers using Anthropic’s API. It enhances efficiency and cost-effectiveness by allowing users to store frequently used contexts and fine-tune model responses. With significant cost savings and improved performance, prompt caching is set to become an essential tool for developers leveraging AI models.
