A user reports slowness in the checkout flow. You open the traces view and search for traces with latency above 2 seconds in the last hour. The query touches thousands of spans across 30 microservices. You wait. The loading spinner turns. 15 seconds. 25 seconds. You start wondering if the query is actually running or if the page froze.

In a traditional trace viewer, the experience is binary: either you have all the results or you have none. For simple queries over short time ranges, this is fine. But for complex distributed traces spanning dozens of services, or for queries covering longer time windows with heavy filtering, the wait becomes a real cost during incident response. Every second staring at a loading indicator is a second not spent investigating.

This post covers how we built traces streaming in Kloudfuse 4.0. Instead of buffering complete query results before rendering, the platform streams trace data progressively as it arrives from the backend. We'll walk through what changes in the user experience, how the streaming delivery works, and the companion performance improvements that make the full system faster.

What Does Progressive Loading Actually Change?

The before/after comparison is straightforward:

Before streaming: You submit a trace query. The backend scans the data store, collects all matching spans, assembles complete results, and sends the entire response. The UI renders nothing until the full payload arrives. For large result sets, this means 10-30 seconds of empty screen.

After streaming: You submit the same query. The backend begins returning partial results as soon as the first matching spans are retrieved. The UI renders those spans immediately while the remainder continues loading. For the same query that previously showed nothing for 25 seconds, you now see the first results in milliseconds and can start analyzing them while the rest loads.

This is not just a UX polish. During an incident, the difference between "I can see the first problematic trace in 2 seconds" and "I need to wait 25 seconds for all traces to load" changes how engineers work. Early-arriving spans often reveal the root cause before the full result set is needed. A span with a 5-second database call or a failed authentication request tells the story without waiting for the remaining 500 spans to arrive.

How Does Streaming Delivery Work?

The backend delivers trace query results via streaming responses. Rather than buffering the complete result set, the query engine begins emitting partial results as spans are retrieved from the data store. The key architectural properties:

Incremental span delivery. Spans arrive at the UI in the order they're retrieved from the storage layer, not in the order they occurred. The UI handles reordering for timeline display while continuously accepting new spans.

Consistent scope. Streaming is not limited to the dedicated traces interface. It applies across dashboard panels and alert evaluation views that consume trace data. A dashboard with trace-backed panels loads progressively, which is particularly noticeable for dashboards with multiple trace queries running concurrently.

Cancellation support. Long-running trace queries can be cancelled directly from the UI. This is critical for a common investigation pattern: start a broad query, see enough initial results to realize the query scope is wrong, cancel, and resubmit with tighter filters. Without cancellation, the engineer either waits for the full scan to complete or refreshes the page and loses their place.

What Companion Performance Improvements Support Streaming?

Streaming delivery improves perceived performance by showing results earlier, but the underlying query engine also got faster across all telemetry streams in Kloudfuse 4.0:

1. Regex alternation optimization. Patterns like service_name =~ "frontend|backend|api" now execute orders of magnitude faster across all telemetry streams, including traces. This is a common pattern for filtering traces by a set of services, and the speedup directly reduces the total data scanning time.

2. Dynamic range query splitting. Large metrics queries that would previously hit series limit errors are now automatically split into sub-queries that each stay within limits. Results are merged transparently. The engineer sees a complete result set without needing to manually narrow the time range.

3. Series limit enforcement at every stage. High-cardinality metrics queries use less memory because series limits are enforced at every processing stage of the query pipeline, not just at the final result. This prevents memory exhaustion on the query nodes and allows more concurrent queries without degradation.

4. Optimized data layout. The underlying storage for log queries was reorganized for faster filtering and aggregation on long time range queries. The storage engine spends less time scanning irrelevant data, which reduces both total query time and time-to-first-result.

Together, these improvements mean streaming is delivering results from a faster query engine. Regex optimization benefits all streams including traces directly. The metrics and logs engine improvements benefit trace investigations indirectly because trace queries share the same underlying query infrastructure. The combination  of "results arrive sooner" (streaming) and "total query time is shorter" (engine optimization) compounds into a significantly better investigation experience.

What Design Decisions Shaped the Streaming Architecture?

1. Server-initiated streaming over client polling. The alternative to server-push streaming is client-side polling: the UI repeatedly requests results at intervals. Polling introduces fixed latency (the polling interval) and wasted requests (polls that return no new data). Server-initiated streaming delivers results at the speed of the query engine, not the speed of the polling interval. For traces where the first useful span might arrive within 50 milliseconds, a 1-second polling interval would add unnecessary delay.

2. Partial rendering over complete assembly. Some trace viewers wait until all spans for a trace are retrieved before rendering the waterfall view. This guarantees a correct layout on first render but sacrifices interactivity. We chose to render partial traces and update them as spans arrive, accepting the visual cost of occasional layout shifts. During an incident, seeing a partial trace immediately is more valuable than seeing a perfect trace 20 seconds later.

3. Cancellation as a first-class operation. Cancellation is often an afterthought in query engines. A cancelled query might stop returning results but continue consuming backend resources. In Kloudfuse's implementation, cancellation stops the query across the pipeline, freeing resources for other queries. This matters in multi-tenant environments where abandoned queries compete with active investigations.

How Does This Work with APM Trace Analytics?

Streaming extends beyond the trace list view into trace analytics (aggregate views of trace data):

Trace error analysis: Error counts and error patterns stream as matching traces arrive. Engineers can see which services are producing errors before the full scan completes.

Latency distribution: The histogram of trace durations refines progressively as more traces are included, but the overall shape typically stabilizes within the first few seconds of streaming.

Service dependency maps: As traces reveal service-to-service calls, the dependency graph populates and builds outward from the most active service paths. For broad queries across many services, this means the map fills in progressively.

What Are the Tradeoffs?

• Layout instability: Progressive rendering means the UI layout shifts as new data arrives. A trace that appears to have 5 spans might grow to 50 as the query progresses. Users who start analyzing a partial result may need to revise their interpretation as the full picture emerges. We mitigate this with clear loading indicators that distinguish "partial results, still loading" from "complete results."

• Result ordering: Streaming results arrive in storage retrieval order, not necessarily in the order most useful for investigation. The UI reorders for display, but the first results to arrive may not be the most relevant. For time-ordered queries (newest first), the storage layout generally delivers recent data first, aligning streaming order with investigation priority. For other orderings, the UI applies sorting on the growing result set.

• Memory pressure on the client: The UI maintains an in-memory buffer of all received spans for a streaming query. For very large result sets (millions of spans), this can pressure the browser's memory. The query engine enforces result limits at the server to prevent unbounded client-side memory growth.

Next Steps

The APM traces documentation covers the trace search interface including streaming behavior. For teams using the MCP Server for AI-assisted trace analysis, the MCP toolsets guide describes the APM toolset capabilities.

How does your team's trace investigation workflow handle large result sets? Are you narrowing queries upfront or waiting for complete results to load? We'd be curious to hear how streaming delivery would change your incident response process.