Agents operate, humans govern: Scale your operations and reduce toil with Azure SRE Agent

What if engineers could spend their time building and optimizing systems rather than maintaining them?

It’s 3 a.m., and the pager goes off. Tabbing between multiple dashboards and diagnostics, the SRE struggles to determine whether what woke them is a real incident, whether they’re the right person to handle it, or whether they need to wake someone else. Digging through monitoring tools, deployment history, incident systems, and team runbooks — and chasing what might be the wrong theory about the root cause — they can’t respond fast enough to stop more customers from being affected.

Or imagine that, by the time the SRE joins the incident bridge, Azure SRE Agent has already analyzed the monitoring data, identified the root cause, and prepared a fix for approval and deployment.

Sanchit Mehta, one of the head engineers for Azure SRE Agent, tells The New Stack that “[Azure SRE Agent] starts analyzing telemetry and correlates things like blast radius, deployment changes, recent changes, any recent rollouts, to try to tell the engineers, ‘OK, this is what is causing it.'” Increasingly, it will even create the PR for that fix.

The support is just as useful during normal working hours. At InEight, correlating telemetry across tens of thousands of Azure resources can take days, if not weeks. When a support ticket reports slow performance without identifying the product, engineers must determine which of the company’s 14 products is affected, then check multiple observability and reliability tools.

InEight shared that, during its first incident using Azure SRE Agent, the agent quickly identified the affected product, traced the performance issue to its root cause, and recommended scaling Redis. The DevOps team had been considering scaling the app service as a temporary fix.

Proactive and in production

This kind of help is becoming the new normal at Microsoft, where more than 3,000 service teams already use Azure SRE Agent to investigate issues, perform root cause analysis, respond to incidents, fix code, enable automatic mitigation, support proactive detection, analyze data, and report at scale. Azure SRE Agent has already handled more than 1.8 million incidents inside Microsoft, many mitigated in minutes.

The team also uses Azure SRE Agent to develop and improve the service itself, with custom agents for code review, deployment, evaluation, and monitoring. This “agent-powered engineering” approach, as Mehta calls it, lets the team take advantage of ongoing advances in AI models. That includes proactively spotting problems, like quota issues that affected deployments, and automatically raising support tickets to resolve them. The agent recently identified the root cause of a change that broke synthetic tests as soon as the change reached the first region, he says.

“It said, ‘OK, this was an upstream PyPI package that broke your dependency; you need to add tests for it; you should roll back immediately; here’s how you should go fix this.'”

Mehta says that kind of proactive monitoring is hard to handle with deterministic queries. “You need a level of intelligence to see when a large production payload is being deployed and if it has the potential to cause degradations.”

For some internal teams, more than half of incidents are autonomously managed by the SRE agent and don’t need any human intervention, adds Shamir Abdul Aziz, lead program manager for Azure SRE Agent, because they’re what he calls “safe” operations and mitigations: a restart, scale-out or rollback of a service, or change order requests escalated by customers.

“The humans did the governance, set up the guidelines, gave some coaching to the agent, and then it went into auto mode to complete the entire workflow,” Abdul Aziz says.

Agents are ready to help

SREs are already drowning in repetitive toil. SREs are already drowning in repetitive toil, and coding agents add to that workload. Agentic operations are now powerful enough to help, Vyom Nagrani, one of the head PMs for Azure SRE Agent, tells The New Stack.

“As code gets written more and more by agents, it’s going to take another agent to operate it,” Nagrani says. “But why wait? If the agent can manage code which other agents write, why can’t it manage code written by humans?”

“As code gets written more and more by agents, it’s going to take another agent to operate it.”

“The reasoning loop has become mature enough that now agents can automatically start figuring out a lot of these complex problems, especially when it comes to correlating across multiple data sources, which has always been the hardest thing for humans to do,” Nagrani says.

Powerful models aren’t enough, though, and homegrown automation won’t have the production-grade governance, verification, evaluation, telemetry, and control a platform can offer.

The state of the art has progressed from prompt engineering to context engineering—which grounds AI in your infrastructure, code, and institutional knowledge — and now to harness engineering. “That is what allows you to run agents at scale, control them, and govern them,” says Abdul Aziz.

“When you combine all these things with being able to verify, audit, evaluate, and get real telemetry and metrics out of the system, where the agent claims it has done something, you can validate that agent’s claim,” Abdul Aziz says.

Instead of a non-deterministic black box that can’t explain its decisions, you can trace and learn from the agent’s reasoning so that you can correct mistakes once, not over and over again. “That’s why companies are willing to adopt it now,” Abdul Aziz says. “Because when you try the same thing ten times, you’re going to get the same output.”

“You don’t just turn on the agent, give it full access, and ask it to solve everything.”

After two years of building enterprise-grade systems that can be trusted, audited, and validated, the next step for cloud-native SRE can be agentic ops with autonomous capabilities — but you still need to know how to adopt it, Abdul Aziz warns. “You don’t just turn on the agent, give it full access, and ask it to solve everything.”

Context and connections

Azure SRE Agent is built for Azure but not limited to the Azure platform. The agent provides native access to Azure services such as Azure Monitor, Application Insights, Log Analytics, and Azure Resource Graph. Connecting the agent to your subscriptions, telemetry data, and source code gives it the operational context and institutional knowledge needed to understand how you work.

Beyond Azure, Azure SRE Agent integrates with engineering and operational tools through managed connectors for Azure DevOps and GitHub, plus MCP connectors that enable access to external knowledge sources such as Google Drive, Confluence, Cursor, Claude Code, and other third-party systems.

Put all that knowledge into Markdown files in a repo, along with the skills and tools agents need to act on your systems (including third-party and on-premises services). That gives you artifacts that agents can version, review, test, reuse, and update.

When you want to dictate how to handle an incident — what to check and in what order, what to post, and even how to format a report — you can create a custom agent, either by using an existing runbook or by working through an incident with an agent and saving that skill. Using agents to improve agents is the shortcut to making Azure SRE Agent more useful the more you use it. Essentially, saving what agents learn during incidents helps improve their future responses.

Guidelines and guardrails

Governance covers identity, role-based access control (RBAC), and tool-access policies. These controls determine which actions are allowed, blocked, or subject to step-by-step approval, and whether an agent operates autonomously or with human review.

What makes governance both flexible and powerful are hooks, based on prompts or deterministic commands, that fire at different stages of a workflow and catch edge cases, such as allowing an agent to drop the index in a SQL database but never drop a table.

Metrics show you whether governance is working. The new live reports show time to mitigation, tool reliability, how often agents act autonomously, and cost per outcome at a glance. InEight’s metrics are typical: an 80% reduction in both incident investigation time and build failure triage time, a 67% reduction in the effort needed to investigate bugs, and an 84% reduction in cost.

To get those results, you need triggers that automatically launch agents instead of waiting for a human to open a chat window.

Bind skills and custom agents to specific alert classes so they can respond to incidents first. Start agents through pipelines, webhooks, or work items to automate delivery workflows. Schedule regular checks, reviews, and audits, and have agents automatically update their artifacts.

Agents operate; you stay in control

By reducing repetitive tasks and technical toil, Azure SRE Agent frees engineers to focus on more interesting and innovative projects. Just as there’s a familiar maturity model for adopting site reliability engineering in the first place, you don’t jump straight into having agents rather than humans handle operations. When you give agents the context about your infrastructure, you can start using them for investigations.

“If you give agents read access to your source code, your telemetry, your resources, the time to get to the root cause is reduced to minutes rather than hours or days,” Abdul Aziz points out. “Every customer starts there.”

Once you’re happy with the answers you’re getting, you can give the agent more permissions while still approving individual steps, he says. “The fixing is easy once you understand the problem. It’s usually changing your configuration, writing a piece of code, or restarting a service.”

“The fixing is easy once you understand the problem. It’s usually changing your configuration, writing a piece of code, or restarting a service.”

As you expand into other operational tasks, refine the agents’ artifacts, metrics, and governance before granting more autonomy: “Things like rolling back a release when we know there was a regression in that release, restarting a service, dropping a corrupt index on a SQL table, or scaling out a service,” Abdul Aziz suggests.

For more complicated issues, agents can deliver the entire fix, ready for approval. The Azure SRE Agent that manages the Azure SRE Agent product looks at exceptions, errors, incidents, Teams conversations, emails, and GitHub issues every night and spits out PRs.

Avoid code review bottlenecks by having agents deploy, test, measure, and include outcomes in the PRs. Use continuous evaluation to build a self-learning system that accurately follows your existing workflows.

“The agent can self-improve because the agent learns constantly,” Abdul Aziz says. “You can configure scheduled tasks to identify which evaluation scores were low and automatically improve the custom agent, custom skills, and even your knowledge documents – because knowledge management is also a toil. The agent can automate all of that.”

The right way to start

Azure SRE Agent now offers a 30-day trial experience with no always-on charges. Make the most of that by learning from some common mistakes:

It’s not magic! Turning on the agent doesn’t mean you don’t have to do DevOps anymore. Don’t treat it as a chatbot or connect it to just your observability system. You need to give the agent the context it needs, the tools to do the job, and intentional triggers that tell it when to act. Otherwise, it may spend effort on low-value work or generate outputs that aren’t grounded in your environment.

Don’t limit yourself to what the agent does out of the box: Customize agent skills, tools, connections, and logic to fit how your organization works, and build custom agents for specific tasks.

Don’t use agents for jobs a single line of code can do: Using them to explore deterministic, structured data for anomalies is an expensive waste of tokens that will only flood the context window when the agent can write that line of code itself. “Orchestrate, don’t calculate,” as Nagrani puts it. If you’re drowning in alerts, use automation to filter the noise and only send alerts that need intelligent analysis to agents.

Don’t stick with what you’ve always done or copy your org chart: The most effective agents have a complete picture of the system, so they need all the context, even if it crosses two teams. That might mean crossing boundaries, coordinating who has expertise and who needs to grant access, or rethinking how the organization works.

“If agents have the right context, they minimize the toil and truly make operations less costly,” lead product manager Deepthi Chelupati points out. That way you can move faster, be proactive, and give engineers more time to innovate and less maintenance work to dread.

Get started today: sre.azure.com

The post Agents operate, humans govern: Scale your operations and reduce toil with Azure SRE Agent appeared first on The New Stack.