How We Built an Enterprise MCP Server for Observability

Published on
Table of Contents
An SRE asks their AI assistant: "What caused the latency spike in checkout?" The assistant needs to check error rates (metrics), search for exceptions (logs), trace the request path (APM), inspect the pods (infrastructure), and correlate the timeline (events). In most observability setups, this means the AI agent needs API keys to five different systems, knowledge of five different query languages, and the ability to correlate results across systems that don't share a common schema.
Most MCP server implementations solve this by wrapping each API as a separate tool. The AI gets a "query metrics" tool, a "search logs" tool, a "get traces" tool. The correlation happens in the AI's context window, consuming tokens and losing precision with every hop. The result is technically functional but operationally shallow: the AI can pull data, but it can't troubleshoot.
This post covers how we built the Kloudfuse MCP Server to go beyond API wrapping. We'll walk through how the server encodes observability domain knowledge into its tool design, how Query Safety Mode prevents AI-generated queries from overwhelming the platform, and how the security model ensures AI agents operate within the same RBAC boundaries as human users.
What Makes an MCP Server "Enterprise-Grade"?
The Model Context Protocol (MCP) is an open standard developed by Anthropic for connecting AI agents to external data sources. Any MCP-compatible client (Claude Desktop, ChatGPT, Cursor, Gemini CLI, custom agents) can connect to any MCP server. The protocol handles tool discovery, invocation, and response streaming.
But implementing the protocol is the easy part. The enterprise requirements are what separate a weekend project from a production deployment:
Governed access. Every query the AI executes must respect the same RBAC policies as the user who initiated the request. A developer who can't see the payments namespace in the UI shouldn't be able to ask the AI to query it either.
Query safety. AI agents generate queries based on natural language prompts. Without guardrails, a poorly scoped prompt can produce a query that scans every metric across every service for the last 30 days at 1-second resolution. That query would consume the same resources as hundreds of concurrent dashboard loads.
Audit trail. In regulated environments, every AI interaction with production data needs to be logged: who asked, what query was generated, what data was accessed, when, and for how long.
Centralized deployment. Engineers shouldn't need to install and configure local MCP servers. The server should deploy once on the platform, scale horizontally, and be accessible to every authorized user through their AI client of choice.
How Does the Kloudfuse MCP Server Connect AI to Observability Data?
The server deploys as a remote service within the Kloudfuse stack, accessible at https://<KLOUDFUSE_URL>/mcp. Enable it with a single Helm configuration:
kf-mcp:
enabled: true
One deployment serves the entire organization. AI clients connect via standard MCP configuration:
{
"mcpServers": {
"kf-mcp": {
"type": "http",
"url": "https://<KLOUDFUSE_URL>/mcp",
"headers": {
"Authorization": "Bearer <KLOUDFUSE_SERVICE_TOKEN>"
}
}
}
}
For interactive use, the MCP server also supports OAuth 2.1 with Dynamic Client Registration (DCR). AI clients register themselves automatically on first connection via RFC 7591, and engineers authenticate through their organization's SSO provider (Okta, Azure AD, Google, SAML). Each user gets a short-lived access token bound to their individual identity, with no shared service account token needed. The full OAuth architecture is covered in our companion post on how we built OAuth for MCP.
What Tools Does the Server Expose?
The server organizes its capabilities into ten toolsets, selectable per connection via URL query parameter (?toolsets=apm,logs):
Toolset | What It Provides |
|---|---|
metrics | PromQL queries, label name/value discovery |
logs | Log search, label discovery, FuseQL and LogQL queries |
apm | Service inspection, dependency mapping, trace search, span drill-down, error surfacing |
alerts | Alert rule search, underlying query retrieval, firing history |
dashboards | Dashboard search, panel queries, folder navigation |
events | Kubernetes event queries with filtering and facet discovery |
infrastructure | Kubernetes object search (pods, deployments, services, nodes) with current status |
pyroscope | Continuous profiling: profile types, flamegraphs, diff comparisons |
rum | Real User Monitoring: applications, events, error groups, latency distributions |
docs | Search and retrieve Kloudfuse documentation |
Different toolset combinations can be assigned to different use cases. An incident-triage agent might use logs,metrics,alerts. A performance-tuning agent might use apm,pyroscope. By default, all toolsets are exposed.
How Does It Go Beyond API Wrapping?
When you query about a service, the server doesn't just return raw API responses. It provides a service-centric view that mirrors how engineers actually troubleshoot:
Current health metrics (throughput, latency, error rates)
Recent traces showing request flows
Logs with errors and warnings in the relevant timeframe
Upstream and downstream dependencies across three layers: service dependencies from distributed tracing, workload dependencies mapping to Kubernetes deployments and pods, and infrastructure dependencies connecting to nodes, storage, and network
Because the tools are designed around entity relationships rather than disconnected APIs, the AI client naturally maintains conversational context across tool calls. "Show me the previous hour" prompts the agent to re-invoke the same tool with a modified time parameter. "What about its dependencies?" triggers the dependency tool using the service entity already in the conversation. The server doesn't need to be stateful because the tool design makes each call self-contained while the AI client tracks the investigation thread.
CTO Ashish Hanwadikar described the design intent: "Our MCP server is not just a wrapper over APIs. It exposes Kloudfuse's full observability model in a way that allows users to troubleshoot issues exactly as they would in the UI: unifying signals across metrics, logs, traces, events, and mapping service and infrastructure dependencies."
How Does Query Safety Mode Work?
This is the feature that no competing MCP implementation prominently describes, and it addresses one of the most practical risks of AI-driven observability: resource exhaustion from poorly scoped queries.
Query Safety Mode validates every AI-generated query before execution:
Bare selector rejection. A query like http_requests_total without any label filter (no namespace, no service, no status code) would scan every instance of that metric across the entire platform. Query Safety Mode rejects it and returns a clear error explaining the violation, so the AI can reformulate with appropriate filters.
Lookback ratio limits. A query requesting 30 days of data at 1-second resolution would produce billions of data points. The safety layer enforces a maximum ratio between the lookback period and query resolution, rejecting queries that exceed the threshold before they reach the storage engine.
Data point caps. Queries exceeding a configurable data point threshold are rejected pre-execution. This catches scenarios where the combination of time range, resolution, and cardinality would produce an impractical result set.
The rejected query returns an error message explaining which constraint was violated and why. This is critical because the AI agent uses that feedback to reformulate the query. A generic "query failed" error would cause the AI to retry blindly. A specific "bare selector rejected: add namespace or service filter" error enables intelligent reformulation.
Query Safety Mode operates at the server level, before queries reach the storage engine. It's both a performance protection (preventing resource exhaustion) and a governance mechanism (preventing overly broad data access).
How Does the Security Model Preserve RBAC Boundaries?
The MCP server supports two authentication methods:
Bearer tokens use Kloudfuse service account tokens for programmatic and headless access. The AI agent operates with the permissions of the service account.
OAuth 2.1 lets users authenticate via their organization's SSO provider (Okta, Azure AD, Google, SAML). Each user gets a short-lived access token bound to their identity. No shared service account token needed.
In both cases, the AI agent inherits the exact RBAC boundaries of the authenticating identity. If a user's team policy restricts them to namespace=frontend, the AI agent querying on their behalf is restricted to the same namespace. There is no privilege escalation through the AI layer.
Stream-specific access controls apply identically. A developer on the frontend team sees logs and traces from their services through the MCP server just as they would through the UI, without accessing other teams' data. A security team with broader access can query across services. The same RBAC policies govern both interfaces.
Every MCP interaction is logged with the user identity, the natural language prompt, the generated query, execution timestamp, duration, and error status. Audit data is self-ingested into the Kloudfuse data store and queryable with the same FuseQL tools used for application logs. Teams can build dashboards showing MCP usage patterns, alert on suspicious access, and satisfy compliance requirements for AI data access governance.
The Design Decision: Intelligence in the Server, Not the Client
The core architectural choice was embedding observability domain knowledge into the server rather than exposing raw APIs and relying on the AI client to correlate.
The alternative, a thin API wrapper, is simpler to build and maintain. But it places the entire correlation burden on the AI's context window. An AI client that needs to correlate a latency spike with log errors, trace failures, and infrastructure events across five separate API calls will consume significant tokens, risk losing context across calls, and produce less accurate correlations than a server that already understands the relationships between these data types.
By encoding troubleshooting patterns (service-centric views, dependency traversal, temporal correlation) into the server's tool design, we reduce the AI client's job to "ask the right question" rather than "orchestrate the right sequence of API calls." The server handles the domain-specific correlation. The AI handles the natural language interface.
The tradeoff is coupling: the server's tool design encodes assumptions about how engineers troubleshoot. If those assumptions are wrong, the tools are less useful. We mitigate this by making toolsets composable and selectable per connection, and by designing each tool to return raw data alongside the correlated views.
Next Steps
The MCP server setup guide covers deployment, client configuration, and toolset selection. The OAuth configuration guide walks through SSO integration for enterprise deployments.
How is your team approaching AI access to production observability data? Are you using MCP servers, custom integrations, or still copy-pasting between dashboards and AI assistants? We'd like to hear what's working and where the gaps are.
