StarTree looks at observability market with Apache Pinot-based real-time engine

StarTree Inc., a company commercializing the open-source Apache Pinot real-time data analytics platform, is setting its sights on the observability tools market and says it’s a target prone to disruption.

Last week, the company demonstrated how StarTree Cloud, a managed Pinot service, can be used as a time-series database compatible with the open-source Prometheus monitoring and alerting toolset to drive real-time observability dashboards using open-source Grafana visualization. engine. By using open source technology, StarTree said it can handle the query and storage components of its observability stack at a much lower cost.

Observability is a discipline and set of tools that provide insight into the internal state and behavior of a system based on the data it produces. The process often involves collecting, analyzing, and understanding three basic types of data—metrics, logs, and traces—to monitor the status, performance, and behavior of complex systems. Observability has become a key feature of cloud-based architectures due to their distributed nature and many moving parts.

Like many early-stage markets, the observability landscape is dominated by a few large companies with proprietary technology stacks and data formats. Many observability vendors also have complex pricing models based on factors such as data ingestion, number of agents, users, features, data storage, and retention periods. costs can reach up to 30% of total infrastructure expenditure and users complain frequently The platforms are very expensive.

These are characteristics of a market poised to be disrupted from below by open source alternatives and low-cost competitors, said Chad Meley, StarTree’s senior vice president of marketing.

pressure from below

“Super-profitable categories are introduced monolithically, but as they are adopted, they force environmental decay,” he said.

Meley cited the example of his former employer, Teradata Corp., which reached a market value of nearly $13 billion in 2012 by selling a tightly integrated suite of hardware and software. As competitive platforms emerge that separate compute, storage, and software, the price premium of an integrated technology stack has become harder to justify. Teradata is still a successful software company, but its market cap is much lower.

The same dynamics apply to observability, Meley said: “But that’s a jump shot because no one has figured out how to assemble the entire stack.” StarTree does not propose to do this, but the company thinks it has a compelling argument for establishing its Pinot-based engine as a cheaper and more functional query and storage engine than those sold by observability leaders.

At the Current 2024 data streaming conference in Austin, Texas last week, he demonstrated how the Pinto-powered StarTree Cloud can be used as a Prometheus-compatible time series database to drive real-time Grafana observability dashboards. Both Prometheus and Grafana are well-established open source standards.

StarTree recently added support for Prometheus and PromQL, a query language that allows the user to select and aggregate time series data in real time. The company also plans to add support for LogQL and TraceQL, which are open-source query languages ​​for logs and traces.

“Then you’ll have an open source-based query and storage engine for three primary types of observability data,” Meley said. “We think we will emerge as the standard.”

Pinot advantage

Six-year-old StarTree, which has raised $75 million in venture funding, thinks Apache Pinot has many features that observability teams will like at a much lower cost than what they currently use. Pinot enables real-time ingestion and querying of data with sub-second response times, even for complex queries on large datasets. It scalable to handle petabyte-sized datasets, supports a large library of plug-in extensions, and integrates with popular analytics platforms such as Apache Kafka, Apache Spark, and Apache Hadoop.

“We are a columnar database, so we ingest data in real time, index in real time, and query in real time,” Meley said. “You get the benefits of the SQL dialect without any lag. You also get the scale that comes with the columnar structure.”

StarTree isn’t ready to announce its entry into the observability market just yet, but “we’re pursuing it aggressively because we see some inherent advantages (of Pinot) and some pretty small product gaps that need to be filled,” Meley said. “We already have some existing customers who want to use us for this use case.”

Picture: Pixabay