The Pros and Cons of High Cardinality in Observability

High cardinality data plays a pivotal role in enhancing observability by providing detailed insights into system performance and user behavior. Its ability to offer granularity across various attributes allows for thorough analysis, slicing, and dicing of data, enabling organizations to delve deep into their operational metrics and event logs. In this blog, we'll explore the multifaceted advantages of high cardinality data, along with the challenges it presents in terms of costs and resource management within observability systems.

Benefits of High Cardinality

The benefits of high cardinality in observability are multifaceted and pivotal for gaining comprehensive insights into system performance and user behavior. Here are some key advantages:

1. Granular Analysis

High cardinality enables detailed granularity across multiple dimensions, supporting comprehensive analysis, segmentation and slicing and dicing of data based on various attributes. For example, in analyzing application performance, trace or span events capture detailed information about individual transactions or requests within a distributed system. High cardinality attributes such as service name, endpoint, user ID, and operation type provide fine-grained dimensions for analyzing the performance and behavior of the system, revealing the various factors influencing its operation. Capturing high cardinality attributes also uncovers unexpected issues (aka "unknown unknowns") during troubleshooting efforts, helping in the identification and resolution of complex problems.

2. Precise Troubleshooting

With high cardinality data, it's easier to pinpoint the root cause of issues or anomalies. High cardinality enables correlation and contextual analysis by incorporating multiple dimensions into the investigation process. Contextual analysis involves examining events within their broader context to understand the sequence of events leading up to a particular outcome or behavior.

For example, consider this trace data:

{traceID=”3ca7188ed2774f2e”, spanID=”7b3f080f9c16292d”, service=”order-service”, operation=”validateOrder”, duration=”50ms”, httpMethod=”POST”, url=”/order/submit”, statusCode=”200", customerID=”xyz789", paymentMode=”credit_card”}

Using this data, when investigating a slowdown in duration, you can analyze the trace or span events by attributes such as service, operation, or customerID to identify the specific service, operation, or user responsible for the degradation. Additionally, you can correlate this span with others using the traceID to see how the request has moved through a distributed system and has faced various bottlenecks.

3. Predictability of Future Behavior

High cardinality data, with its diverse range of distinct values, enhances the precision of statistical and machine learning models, especially in recommendation systems and forecasting applications. By mitigating the risk of overgeneralization, it generates insights grounded in a richer and more diverse dataset. For example, high cardinality data regarding user interactions with a web application can lead to more accurate predictions of future system behavior and potential bottlenecks during certain timeframes.

While high cardinality data presents numerous benefits, its predominant challenge revolves around elevated costs. If the observability system is not optimized effectively, expenses related to data ingestion, analysis, and storage can escalate significantly. These aspects will be examined in the next section.

Cost of High Cardinality

High cardinality can lead to substantial costs, especially for metrics, because of the continuous emission or scraping of metric series at regular intervals, resulting in large data volumes. Even when metric values remain unchanged, sometimes it's essential to record this information because the absence of a series itself can convey important insights or signal an expected state.

The cost implications of high cardinality data need to be addressed across various aspects of observability systems:

• Edge Costs

High cardinality metrics increase CPU, memory, and resource usage in both the originating application and collection agents (e.g., Prometheus, Datadog). In edge applications, relying on events or logs rather than high cardinality metrics also requires more buffer space and resources.

• Network Egress Costs

High cardinality metrics lead to higher egress costs due to the data transfer across the network to observability platforms. These costs are particularly elevated when data is sent to SaaS observability vendors, as opposed to hosting observability in a private cloud environment and managing internal data transfers.

• Data Ingestion Costs

After the data reaches the Observability platform, high cardinality directly impacts the cost of ingestion. As the number of data points increases, so does the demand on CPU usage and memory during ingestion. Scaling up resources becomes essential to handle this increased workload efficiently.

• Query Costs

High cardinality can pose challenges during query time, particularly when performing data aggregates or transformations that process a large number of individual data points to compute the results.

• Storage Costs

Storage concerns arise because high cardinality increases the number of unique data points that need to be stored. Each unique value requires additional storage space, especially if the system needs to maintain indexes or other data structures for efficient querying.

The cost of high cardinality during data ingestion, processing, query execution, and storage requires more computational and storage resources. However, with proper strategies in place, organizations can manage these costs effectively.

We will discuss these strategies in our next blogs.

Frequently Asked Questions

What are the main benefits of high cardinality data in observability?

Three. Granular analysis, because attributes such as service name, endpoint, user ID and operation type let you slice and segment data along many dimensions. Precise troubleshooting, because those dimensions support correlation and contextual analysis down to the root cause. And better predictability, because statistical and machine learning models trained on a richer dataset generalise less and forecast more accurately.

Where do the costs of high cardinality show up?

Across five layers of an observability system: edge costs in the originating application and collection agents, network egress costs when data crosses to the observability platform, data ingestion costs as CPU and memory demand rises with the number of data points, query costs when aggregations process many individual points, and storage costs because every unique value needs space and often index entries.

Why are metrics the most expensive data type for high cardinality?

Because metric series are emitted or scraped continuously at regular intervals, so each unique combination of attributes produces a steady stream of data points. Even when a value does not change it is often still recorded, since the absence of a series can itself signal an expected state or an important condition.

How does high cardinality help with root cause analysis?

Take a span with attributes such as service, operation, duration, customerID and paymentMode. When duration degrades, you can break the spans down by service, operation or customerID to find the specific service, operation or user responsible, then follow the traceID across other spans to see where the request hit bottlenecks in the distributed system.

Why do egress costs rise when telemetry is sent to a SaaS vendor?

High cardinality metrics transfer more data across the network to the observability platform. That transfer is charged as egress, and the charge is particularly elevated when the data leaves for a SaaS observability vendor rather than staying within a private cloud environment where transfers remain internal.