For decades, the observability industry has answered one basic question: Is it running? Agentic artificial intelligence breaks that model. An agent can return a clean response, meet its latency target and throw no errors, yet still give a customer the wrong answer, call the wrong tool or pull from a stale knowledge base.
By every traditional metric, the system is healthy, but the only metric that matters to the business is that it failed. That’s the gap Amazon Web Services Inc. is targeting with Amazon CloudWatch Omni (pictured), which became generally available last week. AWS positions Omni as the next generation of CloudWatch.
It’s app-centric, runs outside the AWS Management Console, is built on OpenTelemetry and embeds AI. This shifts observability from “Is it running?” to “Why did my agent do that?” I’d argue that’s the question standing between most enterprises and production-scale agentic AI. AWS cites an IDC forecast of more than 1 billion deployed agents by 2029. No operations team can manually review that much nondeterministic behavior.
Here are five considerations for information technology and business leaders:
Evaluation is the new monitoring
The most important part of Omni isn’t the dashboards; it’s the evaluation engine. Omni captures every trace and ships with 17 built-in evaluators that score coherence, helpfulness, faithfulness and routing correctness, among other metrics. Teams can compare prompt versions in a playground, build test datasets from production traffic, and automatically catch regressions. Evaluators can also run continuously against live traffic, so quality drift is flagged the same way a CPU spike would be. Latency and error rates can’t tell you whether an answer was right, but scoring can.
Sony is an early adopter. “At Sony, our enterprise-wide agentic AI platform now supports hundreds of proof-of-concept and production workloads,” said Masahiro Oba, senior general manager of the AI Acceleration Division at Sony. “At this scale, observability and evaluation are essential. With Amazon CloudWatch Omni, I can go from a single trace directly to evaluation, AI analysis, comparison or dataset creation.”
The key word in that statement is “hundreds.” Most companies I talk to aren’t stuck on building a single agent. They’re stuck on governing dozens or hundreds, each built by a different team with a different idea of what “good” looks like. Oba also noted that assembling evaluation datasets is often a business-side bottleneck, and that one-click dataset creation from live traces removes that bottleneck.
Getting out of the console is a bigger deal than it sounds
Developers get a native extension for Visual Studio Code, Cursor and Kiro, where traces appear as they run an agent locally, with no AWS account required. Operators get a standalone web experience with single sign-on via existing identity providers such as Okta and Microsoft Entra ID. Both share a single data layer, so the trace a developer debugs is the same one an operator investigates.
AWS acknowledges that its console is built for infrastructure administrators, not for the site reliability engineers, AI engineers and application owners who now carry operational responsibility. Meeting developers in the integrated development environment, where AI coding assistants such as Claude Code and Codex can set up instrumentation, shifts quality work to the point where problems are cheapest to fix.
The unified data layer is the real differentiator
Many startups can trace large language model calls. What’s interesting about Omni is that agent traces, application telemetry and infrastructure signals all live in the same CloudWatch data store. As a result, an investigation can start with an agent receiving a bad tool result, move to an application programming interface error from a capacity-limited service, and end with an exhausted database connection pool.
In most shops today, that’s three tools, three teams and a lot of meetings to connect the dots. AWS DevOps Agent is enabled by default in investigation sessions, correlating signals and maintaining a full investigation history.
Capital One was a design partner. “Capital One operates one of the largest observability footprints in financial services,” said Parvez Naqvi, managing vice president of cloud platform and resilience engineering at Capital One. “As a design partner for Amazon CloudWatch Omni, we helped shape a single AI-powered observability solution that will give our engineers topology-aware intelligence and natural-language querying across all telemetry from a single surface, with full data ownership through OpenTelemetry.”
For heavily regulated industries, such as banking, data portability is critical to success, and that requires data ownership. Captured investigation history is also underrated, since in regulated industries it’s audit evidence of how an AI incident was handled.
Open standards lower the lock-in bar but don’t eliminate it
Instrumentation runs on OpenInference and the AWS Distro for OpenTelemetry, whether agents run on AWS or in other clouds. Omni supports LangChain, LangGraph, CrewAI, the OpenAI Agents SDK, Strands and the Vercel AI SDK, as well as third-party evaluators such as DeepEval. Amazon Bedrock AgentCore agents receive Omni automatically. Azure ingestion is supported today, with deeper multicloud coverage coming.
That openness is necessary in a crowded field. Datadog, Dynatrace, New Relic, Grafana Labs and Splunk are all expanding into agent observability, as are large language model-focused tools such as LangSmith and Arize AI.
OpenTelemetry makes the data portable, but the intelligence layer isn’t. The topology, evaluators, investigation history and DevOps Agent all run on AWS. Companies already heavily invested in AWS will see Omni as the natural default. Those running mature Datadog or Splunk estates across multiple clouds will likely use it for agent development and evaluation, while keeping their observability system of record where it is.
Pricing is built for adoption, so watch the telemetry bill
The IDE extension is free. Customers pay for the telemetry they send and store. Dashboards and alerts are free, and queries up to five times the monthly ingestion volume are included. Eligible accounts receive a 30-day trial and $1,000 in OpenTelemetry ingestion credits.
The catch is that agents are chatty. Every prompt, model call, tool invocation and sub-agent handoff generates a span. Multiply that across hundreds of workloads and continuous evaluation, and ingestion costs can outpace the AI budget that created them. DevOps Agent is also priced separately.
What this means for buyers
Omni is a comprehensive agent observability platform that addresses the trust gap that keeps agents in pilot mode. IT leaders should:
- Define “good” before you buy the evaluator. Built-in scoring only helps if your business owners have documented what a correct, compliant and helpful answer looks like for each agent. Most haven’t.
- Standardize instrumentation now. Put every agent pilot on OpenTelemetry, regardless of the backend. That keeps your options open and makes future platform decisions much easier.
- Model telemetry costs at production scale. Set policies for sampling, retention and evaluation frequency before agents go live, not after the first bill arrives.
- Decide where your system of record resides. If AWS is your primary cloud, Omni is a strong default choice. If you’re multicloud with an established observability platform, consider Omni for development and evaluation, and integrate it rather than replacing your existing setup.
- Treat investigation history as part of governance. Integrate Omni’s captured investigation trail into AI risk and compliance processes, especially in regulated industries.
The industry spent a decade learning to observe distributed systems. Agentic AI requires observing decisions, not just systems, and AWS aims to own that layer for its customers. The companies that get the most out of it will be those that treat evaluation as an operational discipline, not a feature to be switched on.
Zeus Kerravala is a principal analyst at ZK Research, a division of Kerravala Consulting. He wrote this article for SiliconANGLE.