Agents are only as good as the telemetry underneath them

The question on stage

In September I joined a panel at The Next Endeavour 2026, hosted by Imagination in Action at Google Bay View. The title was blunt: Why 95% of AI Pilots Fail.

Paul Baier of GAI Insights moderated. Next to me were Andrew Lau, CEO of Jellyfish, who sees how AI changes engineering teams, and Rahul Todkar, CEO of Nasiko, who builds a runtime for running agents. I was last in the lineup, which fit. If you follow the software lifecycle from code to team to runtime, production observability is the last stop. It’s where pilots either survive or quietly die.

Paul asked each of us for one message. Mine was simple: agents are only as good as the telemetry underneath them.

Here’s what I meant, and what I’ve seen since.

The 95% number isn’t the scandal

MIT’s NANDA initiative reported that about 95% of enterprise generative AI pilots had little or no measurable impact on the P&L. Gartner predicts more than 40% of agentic AI projects will be canceled by the end of 2027 over cost, unclear value or weak risk controls.

I don’t read those numbers as AI failing. I read them as a candy store. For two years everyone grabbed everything, because everything looked exciting and we thought we could finish it all at once.

That phase is ending. Engineering discipline is setting in. Teams are bringing the software development lifecycle to agents. But when I talk to customers, three problems keep stopping agents short of production. All three are visibility problems.

Reason 1: You can’t fix what you can’t see

On the panel, Rahul told a story I hear constantly. His team built an agent in a couple of hours. Something went wrong. It took three weeks to debug.

Three weeks. You can’t run production like that.

For regular applications, we’ve solved this. We know how to run a service at three or four nines. We know which metrics, logs and traces to keep and which alerts matter. Agents are new, and most teams are running them with a fraction of that visibility. In Cleanlab’s 2025 survey, fewer than one in three teams with agents in production were satisfied with their observability and guardrails. In LangChain’s latest survey, 89% have some observability, but only 62% can trace an agent step by step.

I learned this again recently, from the other side. Late one night I was chasing Kafka timeouts after we moved one of our own internal clusters to a new cloud. I worked through it with an AI agent. It was genuinely useful. It found a disk pinned at 100% utilization while moving almost no data, and traced the timeouts back to it.

But it was useful for one reason: it had metrics, logs and traces to reason over. Take those away and it would have guessed, confidently. That’s the whole point. The agent was only as good as the telemetry underneath it.

Reason 2: Agents are self-pleasing

The second reason is what I call the self-pleasing agent. It runs on my desktop. It passes my eval. Of course it solves the problem.

Then it meets the real world, and the real world is messy.

Datadog wrote about this while building evals for their own SRE agent. One of their key findings was that simulated environments have to be noisy, full of the distractions a real system has, or the results are too optimistic. The numbers say most teams haven’t caught up. In LangChain’s survey, 52% run offline evals, but only 37% evaluate agents against real production traffic.

One of our healthcare technology customers showed us what closing that gap looks like. Before they ever talked to us, they had built their own agent tooling. The workflow their engineers used most wasn’t a dashboard. It was comparing a trace from a development run against a known-good production trace, side by side, to see exactly where the agent’s path diverged.

That’s the right instinct. An eval tells you whether the answer looked right. A trace tells you how the agent got there, and whether it will get there again.

Reason 3: Design for the human, not for autonomy

This is the one I feel most strongly about.

A year ago, everyone in our industry was talking about autonomic systems. Production would run itself. Now people are running into the limits of what LLMs and agent harnesses can do.

So design the system with the human in mind. Give people real control. Sometimes that’s a kill switch. Often it’s smaller: a way to nudge the agent in the right direction, because the agent didn’t have the full context. Usually it doesn’t.

The teams actually shipping agents already work this way. A study of agents in production found that 68% run ten steps or fewer before a human steps in.

Our customers in healthcare and financial services have taught us to go slow on purpose. Agents start with read-only access. Once people trust what they see, the humans set the guardrails, and only then does an agent really become an agent.

It’s how we built Dexter, our AI SRE. Your monitoring detects. Dexter investigates. Your engineers decide.

Who is responsible when an agent goes wrong?

This is the question I always ask, and it got the strongest reaction on stage. When an agent does something it shouldn’t, as one of my fellow panelists put it, someone’s head is on the block. “I spun up an agent and it did this” is not an answer.

Systems are complex. We’ve seen this before, and agents won’t be different. Every team running agents needs an owner and the controls to back them up. Here’s what that owner’s telemetry should answer, every day.

Where should agent telemetry live?

Here’s the part people skip. Agent telemetry isn’t like CPU metrics. It holds prompts, tool calls and outputs, and in healthcare or finance that means patient details and account data.

Shipping all of that to someone else’s cloud so you can debug your agents is a bad trade. It’s also expensive, because agents generate far more data than the services around them. That’s why Kloudfuse runs inside our customers’ own cloud as Self-SaaS, a managed BYOC model. The data never leaves their environment, and they keep open standards like OpenTelemetry so they’re never locked in.

The same logic is why I care about open models. I’ve been waiting for their Linux moment. Right now “open-weight” too often gets read as “from China,” and open models’ share of enterprise usage actually fell last year, to 11% from 19%, according to Menlo Ventures. Linux won because Google, IBM and Red Hat put real weight behind it. Open models need the same from US big tech. When they get it, more teams will run models inside their own walls, and their telemetry should be right there with them.

What I’d tell a team starting today

1. Instrument before you scale. If you can’t trace one agent step by step, don’t run ten.
2. Test against production, not your desktop. Compare real traces, not just eval scores.
3. Start read-only. Let trust, not a roadmap, decide when an agent gets more access.
4. Name an owner. Every agent in production needs a person who answers for it.
5. Keep the data in your environment. Your agents’ telemetry is some of the most sensitive data you have.

The teams that win this won’t be the ones with the most agents. They’ll be the ones that can see what their agents are doing.

FAQ

Why do AI agent pilots fail to reach production? Three common reasons: teams lack the telemetry to debug agents, evals don’t reflect messy real-world conditions, and systems aren’t designed for human control. All three are visibility problems.

What is agent observability? Agent observability is the ability to see what an AI agent did, which tools it called, how many tokens it used, and why. It extends metrics, logs and traces to agents.

What is the difference between observability and monitoring? Monitoring tells you whether a system is running. Observability tells you why a running system is behaving strangely, using telemetry such as metrics, logs and traces.

Should AI agents have write access in production? Not at first. Start agents read-only. Once people trust their behavior, let humans set guardrails and expand what the agent can do.

Where should agent telemetry be stored? Ideally inside your own environment. Agent telemetry contains prompts, tool calls and outputs that often include sensitive customer data.