When a traditional service fails, the failure is usually obvious. An HTTP 500 error. A timeout. A crash. The metrics turn red. The alerts fire. The on-call engineer wakes up. The system is designed to make failures visible.

AI systems do not fail this way.

When an LLM-powered service degrades, it often continues returning HTTP 200 responses. The API contract is satisfied. The latency may be within normal bounds. The service is technically running. But the responses are wrong. The model is hallucinating. The retrieval pipeline is returning stale documents. The embedding quality has drifted. The agent is looping through the same tool calls without converging on an answer.

The service is green in your traditional monitoring. The experience is broken for your users.

This is the fundamental problem with treating AI monitoring as a separate category from application monitoring. Traditional observability tells you the service is up. AI monitoring tells you the service is working correctly. But there is a third dimension that gets lost when these systems are disconnected: the underlying cause of an AI application failing may not be the model at all. It could be infrastructure and platform issues: a Kafka or MSK pipeline dropping messages, a Postgres or Pinecone database returning stale results, or physical resource constraints like CPU contention, memory pressure on nodes, or GPU saturation. 

If AI telemetry, application traces, and infrastructure signals live in different products with different dashboards and different data pipelines, the gap between them is where production incidents hide. Unified observability across AI applications, platform services, and infrastructure is not a nice-to-have. It is the only way to diagnose failures that cross these boundaries.

The Industry Built AI Monitoring as a Silo

When LLM applications started moving to production in 2023 and 2024, the observability industry responded by building dedicated AI monitoring products. This made sense as a first step: the telemetry was new (prompts, tokens, model identifiers, completion quality), the users were new (ML engineers, AI platform teams), and the questions were new (is the model hallucinating? What is the cost per token? how does the retrieval quality compare across embedding models?).

The result was a generation of AI monitoring tools that exist alongside traditional observability but do not integrate with it.

1. Datadog's Agent Observability (formerly LLM Observability) has made meaningful progress toward APM integration. Their documentation now describes correlating agent behavior with backend services, infrastructure metrics, and user sessions within the same trace ID. That is a genuine step toward unification. But Agent Observability remains a separately purchased product: a free tier of 40,000 LLM spans per month, and Pro starting at $160 per month for 100,000 spans. This sits on top of APM ($31 to $40 per host), infrastructure monitoring ($15 to $23 per host), and log management ($0.10 per GB plus $1.70 per million indexed events), each billed independently. The technical correlation is tightening, but the commercial model still treats every observability dimension as a separate billing surface, making total cost of AI monitoring difficult to forecast.
   
   

2. New Relic’s AI Monitoring is more tightly integrated with APM than its original category framing implied. New Relic’s documentation shows AI agent monitoring enabled through the existing APM agent, with agent and tool activity visible through end-to-end monitoring workflows, including trace-centric investigation. At the same time, AI workloads can materially increase observability data volume, and New Relic’s pricing remains usage-based around data ingest, with platform access and compute-based options depending on plan. That means the technical experience is increasingly unified, even if AI adoption can still expand cost exposure through higher ingest and heavier query usage.
   
   

3. Grafana’s AI Observability, introduced in 2025, is built on OpenTelemetry and delivered inside Grafana Cloud through a dedicated plugin and integration experience. Grafana documents pre-built dashboards and views for AI observability, including analytics for activity, latency, errors, tokens, cost, and conversation-level drilldowns, and its integration reference also lists dashboards for GPU monitoring, MCP observability, and vector database observability. This makes it more connected to the broader Grafana observability stack than a standalone AI tool would be. But it still enters the platform as a distinct integration layer, so AI-specific investigation typically starts from dedicated AI views rather than the same default surface as core APM.

The pattern is evolving, but it is not fully resolved. Datadog, New Relic, and Grafana have all moved toward tighter linkage between AI telemetry and traditional observability through trace correlation, shared investigation paths, built-in dashboards, or entity-level views. But AI observability still commonly shows up as a separately named capability, a separately configured experience, or a separately monetized layer on top of the existing platform.

The real question is no longer whether AI telemetry is present somewhere in the platform. It is whether model behavior, application performance, and infrastructure health can be investigated as one operational system without forcing teams to cross product, pricing, or workflow boundaries.

Why This Architecture Fails During Incidents

Consider a production scenario that is becoming increasingly common. A customer-facing application uses an LLM to generate personalized responses. The application receives user input, passes it through an embedding model to retrieve relevant context from a vector database, sends the context and prompt to an LLM, receives the completion, and returns it to the user.

The response quality starts degrading. Users report answers that are factually wrong or irrelevant. Your SRE team starts investigating.

In a siloed architecture: The SRE opens the application monitoring dashboard. Response times look normal. Error rates are low. HTTP status codes are 200. The application appears healthy. They check the AI monitoring product. Token latency is elevated. But is the problem in the model inference? The embedding computation? The vector database query? The network path between the application and the model endpoint? To answer these questions, they need to cross between two products, correlate data manually, and reconstruct the request flow from fragments.

In a unified architecture: The SRE opens the trace for a degraded request. In that single trace, they see the HTTP request span, the embedding model call with latency and token count, the vector database query with the retrieval results and latency, the LLM call with the prompt, response, token usage, model identifier, and inference time, and the response back to the user. Every step of the request lifecycle, from the load balancer to the model completion, is in the same trace. The SRE can see immediately that the vector database is returning stale documents because its index has not been refreshed since a data pipeline failed six hours ago. The model is fine. The embedding is fine. The retrieval is stale.

That investigation takes minutes in a unified trace. It takes hours when the data is spread across two products.

LLM Monitoring Belongs in the Trace, Not in a Silo

We made an architectural decision early in building Kloudfuse's AI observability capabilities: LLM monitoring would not be a separate product. It would be a native property of the APM trace.

When an application instrumented with Kloudfuse makes an LLM call, the telemetry is captured as attributes on the trace span:

• Prompt content: the full prompt sent to the model, including system instructions, user input, and retrieved context

• Response content: the model's completion, including any structured output or tool call responses

• Token metrics: input tokens, output tokens, total tokens, and cost (when model pricing is configured)

• Model metadata: model identifier, provider, API version, and configuration parameters

• Performance: inference latency, time to first token (for streaming responses), and total processing time

These attributes sit on the same span as the HTTP request, the database query, and the downstream service call. They are queryable through the same query interface. They appear in the same trace waterfall. They are governed by the same RBAC policies and protected by the same FIPS 140-3 validated encryption.

For AI workloads that involve multiple model calls, agent loops, or tool-use chains, the trace captures each step as a child span. The resulting trace shows the complete execution path of the AI workflow: which agent was called, which tools it invoked, which models it queried, what decisions it made, and how long each step took. When the workflow involves non-AI operations — database writes, API calls to external services, cache lookups — those operations appear in the same trace alongside the AI steps.

The Data Sensitivity Question

There is a second reason LLM monitoring architecture matters, and it has nothing to do with incident response speed.

LLM telemetry is among the most sensitive data that an observability platform will ever process. Prompts contain user inputs. They may contain documents, questions about internal systems, personal information, or business-sensitive queries. Model responses contain the AI system's outputs, which may reflect internal business logic, proprietary reasoning, or sensitive generated content. Token-level metrics reveal usage patterns that can indicate which features are used and how.

When this telemetry flows through a SaaS vendor's dedicated AI monitoring product, it is processed and stored on the vendor's infrastructure. For many organizations, especially those in regulated industries, this creates the same data residency and sovereignty concerns that apply to all sensitive operational data — compounded by the fact that LLM telemetry is often more sensitive than traditional metrics and logs.

Kloudfuse's approach addresses this directly. LLM telemetry flows through the same data pipeline as all other observability data. It is stored in the same data store, in the customer's VPC. It is encrypted by the same FIPS 140-3 validated module. It is scrubbed by the same data scrubbing rules that protect PII and sensitive content across all five telemetry streams. It never leaves the customer's infrastructure.

This is not just a deployment preference. For organizations deploying AI in production, it is increasingly a requirement.

The Architecture Decision That Defines the Next Five Years

The observability industry is at a fork. Down one path, AI monitoring continues as a separate product, bolted onto existing platforms, sold as an add-on, and operated through a different workflow than the rest of the observability stack. Down the other path, AI telemetry becomes a native property of the application trace, monitored with the same tools, queried through the same interface, and governed by the same security model as every other production signal.

The first path is easier to build and easier to sell. The second path is harder to build but fundamentally more useful when the AI system fails at 3 AM and your on-call engineer needs to understand why in minutes, not hours.

We built Kloudfuse for the second path. Because AI systems fail quietly. And when they stop being quiet, the answer needs to be in the trace, not in a different product.

AI-native observability is not a feature. It is an architecture.

Kloudfuse unifies LLM monitoring in APM, enterprise MCP with query safety, and AI-assisted operations in a single platform. Deployed in your VPC. FIPS 140-3 certified.