StarTree, the leading commercial vendor behind the Apache Pinot real-time analytics data store platform, made several major product updates at its annual Real-Time Analytics Summit. These updates aim to make large-scale, real-time data analytics and observability more accessible in the cloud era.
Pinot, led by CEO Kishore Gopalakrishna, was created by a former engineer at LinkedIn. The platform provides a reliable, high-speed data store with specialized indexes and optimization for real-time analytics at scale. Stripe, Walmart, and DoorDash are among the many large enterprises that rely on Pinot. However, Pinot faces competition from other open-source options, such as the StarRocks online analytical processing (OLAP) database.
The core announcements at the Real-Time Analytics Summit revolve around the open-source Apache Pinot project and StarTree’s commercial offerings that build upon and extend Pinot’s real-time analytics capabilities. Notable updates include a new serverless cloud service, native integrations with leading data visualization tools like Grafana and Tableau, the general availability of the ThirdEye observability service, vector search support, and a new cloud write API.
One of the areas where Pinot is seeing increased adoption is in metrics, logs, and traces. Real-time analytics is becoming more widely applicable across various use cases. To support these observability use cases, StarTree is launching ThirdEye as a generally available service. ThirdEye helps with identifying anomalies in real-time and conducting root cause analysis for complex business metrics that traditional monitoring systems struggle with. By leveraging Apache Pinot as its foundation, ThirdEye can compute metrics from data points in real time.
It’s important to differentiate between business metrics and system metrics about IT systems. Traditional monitoring systems are ill-equipped to handle complex business metrics. For example, ride-sharing vendor Uber had all of its system metrics monitored by a PagerDuty system, but this technology couldn’t easily monitor derived and computed metrics like driver supply metrics.
Another highlight at the conference is the release of Apache Pinot 1.1. The new release introduces vector index support, which is particularly useful for large language models (LLM) and generative AI use cases. Vector support has become a core capability for databases, with Google recently announcing that all of its cloud databases would support vectors. Apache Pinot 1.1 adds support for Hierarchical Navigable Small Worlds (HNSW) graphs to enable vector index search.
Gopalakrishna emphasized that adding support for vectors in Pinot was not a difficult task since the technology already had multiple types of indexes. Pinot’s strength lies in its ability to provide low latency even at high concurrency, thanks to its specialized indexes for different data types like geo, text data, JSON, and numerical data.
Overall, StarTree’s product updates and the advancements in Apache Pinot showcased at the Real-Time Analytics Summit demonstrate how companies are responsibly integrating AI in production. These developments make large-scale, real-time data analytics and observability more accessible in the cloud era, catering to various use cases and providing specialized solutions for complex business metrics.