High Cardinality Metrics: Avoiding Common Pitfalls

In our last blog posts on high cardinality data in observability, we discussed the fundamentals of observability and its growing importance. Now, we'll delve deeper into understanding how high cardinality manifests across all four types of observability data streams: metrics, events, logs, and traces.

In this blog, we will focus only on metrics as the cardinality behaves differently in metrics vs. other types of observability data. In the next blog, we will cover events, logs, and traces.

Metrics are often numeric values, such as counts, averages, or percentages, computed over a set timeline to present overall performance or behavior of a system. For example, a metric could measure the number of page views per minute or the average response time of a server.

Metrics play a critical role in detecting anomalies, outliers, and performance trends of a system over time. In contrast, events, logs, and traces capture discrete system occurrences such as errors, startup/shutdown events, and system state changes, emphasizing qualitative details vs. continuous numerical measurements across fixed time intervals typical of metrics. As a result, high cardinality can be more pronounced in metrics and requires more careful management compared to events, logs, and traces.

In metrics monitoring, the frequency of data capture intervals and the inclusion of various attributes or dimensions associated with the metric elevates its cardinality. For instance, monitoring CPU usage per second and correlating it with pod or container IDs significantly increases data complexity and cardinality. Tracking these details every second, rather than less frequent intervals like every hour, further increases the volume.

While high cardinality data has the detailed granularity and richness to provide accuracy and precision in pinpointing exact system issues, it can also lead to challenges such as inflated storage requirements and higher computational overheads during data aggregation and analysis.

Balancing data richness with resource efficiency is crucial. In future posts, we’ll discuss strategies for achieving this balance. Meanwhile, here are some common pitfalls to avoid to prevent unnecessary cardinality accumulation:

1. Improper Instrumentation

Improper instrumentation occurs when tags or attributes that are not used in dashboards or alerts are included in metrics. For example, adding an IP address to every request, irrelevant for monitoring purposes, can increase cardinality unnecessarily.

2. Histogram Buckets

Histograms are used to track the distribution of values, such as response latencies, within different buckets or bins. Using too fine-grained histogram buckets can lead to high cardinality, especially when tracking percentiles for metrics like response latencies. Therefore, it's crucial to design histogram buckets based on the expected value range and required precision, as adding unnecessary buckets will only increase cardinality.

3. Transient Environment Tags

In dynamic environments like Kubernetes, monitoring metrics often include tags related to infrastructure or environment variables. However, poor choice of tags like container IDs or pod IDs can significantly contribute to high cardinality, particularly in environments with short-lived containers or pods that are frequently created or destroyed. This issue is particularly pronounced in dev/test environments where CI/CD workflows continuously generate new ephemeral resources to test various scenarios. To address these challenges effectively, it is advisable to utilize stable identifiers such as job names or service names for tracking purposes, rather than transient tags like container IDs that can rapidly become obsolete.

By addressing these sources of high cardinality through proper instrumentation, appropriate histogram buckets, and the use of relevant infrastructure and environment tags, organizations can effectively manage cardinality and gain accurate analysis and insights without unnecessary complexity.

Stay tuned for our next blog, where we will dive into managing high cardinality across events, logs, and traces.

Frequently Asked Questions

Why does high cardinality affect metrics more than logs or traces?

Metrics are numeric values computed continuously across fixed time intervals, so every unique set of attributes becomes a series that is sampled again and again. Events, logs and traces record discrete occurrences with qualitative detail. The combination of capture frequency and attached dimensions is what makes cardinality more pronounced in metrics.

What counts as improper instrumentation in metrics?

Attaching tags or attributes that no dashboard or alert ever uses. The example in the post is adding an IP address to every request metric: it is irrelevant for monitoring purposes and multiplies the number of series for no analytical return.

How do histogram buckets increase cardinality?

A histogram tracks the distribution of values such as response latencies across buckets or bins, and each bucket is its own series. Buckets that are too fine-grained, especially when tracking percentiles, add series without adding useful precision. Design buckets around the expected value range and the precision you actually need.

Which tags should I avoid on Kubernetes metrics?

Transient identifiers such as container IDs and pod IDs. In environments where pods are created and destroyed frequently, and especially in dev/test environments driven by CI/CD, each new resource adds a fresh tag value. Use stable identifiers such as job names or service names instead.

Does collection frequency change the cost of high cardinality metrics?

Yes. Capturing a metric such as CPU usage every second, and correlating it with pod or container IDs, produces far more data than capturing it hourly. Frequency multiplies the volume of every series, so it compounds whatever cardinality the tags already create.