Managing Cardinality in Events, Logs, and Traces

In observability, metrics, events, logs, and traces are common types of data that are collected and analyzed. In our last blog, we covered high cardinality in metrics. In this blog we will provide a comprehensive overview into managing high cardinality effectively across events, logs, and trace data streams.

Events

Events, in contrast to metrics, are individual occurrences that are logged or recorded as they happen, rather than being continuously structured into a time series. Therefore, the cardinality of events tends to be more manageable.

Each event typically contains various attributes or properties that describe it, such as a timestamp, user ID, action performed, and any relevant contextual information. For example, in a web application, each user interaction (like clicking a button or submitting a form) can be tracked as an event.

High cardinality in events means that there are many distinct values for certain attributes (e.g., user ID) within the events. High cardinality in events can be useful because it allows for detailed analysis and segmentation of data, where you can drill down into specific subsets of events based on these attributes (e.g., request delay by user ID) to gain insights into user behavior, patterns, and anomalies.

Logs

Log observability is crucial for monitoring and troubleshooting applications and systems. Similar to events, each log entry represents a discrete occurrence or message generated by an application or system component. High cardinality in logs happens when various fields within logs (or the combination of them) result in a large number of unique values.

For example, consider a scenario where you have a web server that logs each HTTP request to a website. Each log entry contains various fields such as User-Agent, IP Address, and Request Path, contributing to high cardinality data:

• User-Agent: This field contains information about the browser or user making the request. There can be thousands of different User-Agent strings, each representing a unique combination of browser type, version, operating system, and so on.
• IP Address: If each request also logs the IP address of the user making the request, it could lead to millions of unique IP addresses accessing the server.
• Request Path: The URL path typically includes query parameters requested by the user for a specific search. This can lead to high cardinality attributes as each URL path is unique.

It's important to note that while individual logs are used to pinpoint the root causes of application or system failures, analyzing the cardinality of log entries over time can reveal patterns that signal anomalies or deviations from normal operations. For example a sudden spike from a specific IP address could indicate unusual or perhaps suspicious behavior. As log data trends over time, its increasing cardinality poses challenges for storage and processing.

Traces

Traces typically represent complex interactions across distributed systems or microservices. Each trace may include multiple spans or events, each with its own set of attributes. For example, traces often involve capturing unique identifiers such as transaction IDs, session IDs, or user IDs. If these identifiers have a wide range of distinct values (high cardinality), it results in a large number of unique trace entries. Managing and storing numerous unique traces can quickly escalate storage requirements and computational overhead.

Managing high cardinality requires strategies that balance thorough data collection for diagnostics with efficient storage and processing. Depending on the data type and analysis requirements, diverse approaches such as aggregation, reduction of unnecessary details, smart indexing, and other techniques can be implemented.

In our upcoming blog posts, we'll explore effective strategies for optimizing observability data analysis and storage.

Stay tuned!

Frequently Asked Questions

Why is cardinality in events more manageable than in metrics?

Events are individual occurrences recorded as they happen, not values continuously structured into a time series. A high cardinality attribute such as user ID still appears on every event, but it does not spawn a stream of regularly sampled series, so the volume grows with activity rather than with time.

Which log fields most commonly cause high cardinality?

In a web server access log, the User-Agent string, the client IP address and the request path. User-Agent alone can take thousands of distinct values, IP addresses can run to millions, and request paths that embed query parameters are often unique per request.

Can analysing log cardinality over time reveal anomalies?

Yes. Individual log lines pinpoint the root cause of a failure, but watching how the cardinality of log fields changes can expose patterns that deviate from normal operation. A sudden spike in entries from a single IP address, for example, can indicate unusual or suspicious behaviour.

Where does high cardinality come from in distributed traces?

From the identifiers that spans carry, such as transaction IDs, session IDs and user IDs. A trace spans many services and each span has its own attributes, so identifiers with a wide range of distinct values produce a large number of unique traces, and storage and compute requirements escalate with them.

What strategies help manage high cardinality across these data types?

The post names aggregation, reducing unnecessary detail and smart indexing, chosen according to the data type and the analysis you need to support. The goal is to keep enough detail for diagnostics while holding storage and processing within reasonable bounds.