Before we start: I’m in San Francisco this week and also next week. On Wed, 4 November, I’m hosting The Pragmatic Engineer Unplugged at the iconic SFJazz Club, featuring a rare conversation with Diogo Almeida (creator of Jev & co-inventor of ChatGPT), and Bryan Cantrill (cofounder and CTO of Oxide). It’s free to attend with registration, sign up here.
In May 2025 at Microsoft’s annual BUILD conference, the tech giant introduced “Azure SRE Agent” with an onstage demo in which an alert fired and the AI SRE Agent went to work on identifying the root cause, and then shipped a fix, all without having to wake up a human dev oncall.
Watching this demo, the whole thing felt artificial to me, and I failed to imagine any engineering team at Microsoft – or anywhere else! – letting an AI make production decisions with no human oversight that could potentially cause a worse outage. And sure enough, after last year’s BUILD I heard no more about “SRE agents” at Microsoft or elsewhere – until recently, that is.
Now, I do hear it being mentioned: OpenAI built an AI SRE bot called Sevbot, and last month, an engineer at ING bank told me they’re working on their own, in-house AI SRE. Just as visibly, at tech conferences I’ve attended, AI SRE vendors’ stands have been popping up in numbers.
To find out what’s going on, we reached out with Jessica to three of the most popular AI SRE startups/AI incident platforms to hear directly from them: Rootly, incident.io, and ResolveAI (note: I’m an investor in incident.io). Thanks for all the input!
Today, we cover:
**Why is AI SRE “hot”?**Incidents and pages are trending upward, thanks to AI agents speeding up at shipping code to production. It only makes sense to invest in better oncall tooling.**What does “AI SRE” mean?**It’s mostly about doing incident investigation much faster, and inside of Slack. Whether “AI SREs” should make code changes is debated.**Building an investigation agent.**Fanning out multiple agents is common, as is the practise of engineers steering them in Slack or Teams. A “context layer” with a knowledge graph and memory is used for better investigation outcomes. Several tech companies built their own investigation agents instead of relying on vendors.**Making agents accurate.**Helpful approaches include verifying incident hypotheses with adversarial agents and a mix of models, as well as running evals and backtesting. However, measuring “accuracy” of AI SREs is tricky and highly specific to each company, and even engineering team.**Making agents fast & affordable.**Everyone wants fast, cheap, and accurate AI SREs, but there are tradeoffs. Model routing, lean prompts, and tracking cache hit rate are ways to engineer a system with the right tradeoffs for your constraints.**How long until autonomous agents?**The fully autonomous agents demoed on conference stages are still not a reality, but AI agents triaging pager calls – and silencing unimportant ones – is becoming more common. Interestingly, there’s not been much change in this area from how Meta operated its own systems with AI two years ago.**What’s changing for on-call engineers?**A new category of AI-related outages is appearing, and engineers should eventually get hit with fewer pages, but the incidents that do reach engineers will be hard. There’s a deskilling risk, and it could be smart to train oncall with incident response simulations like those fromUptime Labs.
As usual, I have no affiliation with vendors mentioned in this article except for my investment in incident.io – mentioned above. I have not been paid to mention any company.
A year back, there was little to no buzz about AI SREs, but that’s all been changing.
**The main driver seems to be that there’s a lot more code deploys. **Almost all tech companies have seen their PR frequency double in a year due to AI agents. Let’s do some basic “reliability math”:
i = c x p:
ncident rate = [
IChange rate] x [Probability]
C= number of changes per unit timeP= probability a single change introduces failure
This says that if we have few deployments (low change rate) and high confidence that each deploy works (low probability of failure), then that is a pretty stable system. But with AI agents, deploy frequency is up and the incident rate is likewise increasing at most companies.
We know that C (the number of changes in a given timeframe) is up twofold or more on last year at many companies, and will probably climb higher. Data from Rootly shows that pages are also up from a year ago, and outside-business-hours pages have nearly doubled:
Broadly, there’s two approaches to lowering P (incident failure rate probability):
Better pre-deploy verification: more testing & verification loops
Learn from incidents: make the system more resilient
Most AI SRE vendors are well placed to learn from incidents and act upon them. For example, here’s Rootly’s high-level strategy for more resilient systems:
“AI SRE” looks on its way to becoming the agreed term for AI-augmented oncall tooling, even though we’ve not spoken to a single person who likes it – and we agree.
**AI SRE vendors themselves don’t like it. **When we talked with incident.io, Resolve, and Rootly for this article, the reply was that, unfortunately, they’ve not come up with anything better and so this name has come to stick instead. Alternative names proved less sticky even though they’re less inaccurate, like “autonomous reliability agents”, “automated remediation agents”, and “automated incident investigation agents.”
A reason “AI SRE” is less than ideal is as simple as the fact that a site reliability engineer (SRE) does a lot more than just deal with oncall alerts! They work on making a system more reliable in its entirety and a good chunk of that work happens before and after outages.
Despite all that, this article is going with the flow and using “AI SRE” because there’s not much choice, while covering oncall and incident handling tools that are AI-augmented/AI-powered.
We cover more on the SRE profession and reliability engineering in a deepdive with Dave Connor, who worked at Google when the term ‘SRE’ was coined.
**AI SRE tools mostly help with the investigation phase of incidents. **Tools like Rootly, incident.io, and ResolveAI are most helpful after an alert has fired and an engineer starts investigating. Pre-AI, this investigation phase involved:
Alert details:inspect the alert and try to figure out what it meansDashboards: check custom dashboards to confirm whether there’s an alert, which systems are impacted, and where the root cause investigation should start, if soRunbooks: consult runbooks and follow the steps to remedy the issue, if it’s a common oneLogs/observability: look through the logs of observability systems like Datadog, Sentry, Honeycomb, etc
These days, AI tools have gotten pretty good at all these tasks if they have sufficient data access. Of the three vendors we talked with, all do their investigations with AI. When building in-house AI SRE tools, starting at the investigation phase is a no-brainer:
**Most AI SRE tools speed up the investigation of an incident. **incident.io’s agent orchestrator does something similar to when an oncall engineer opens eight tabs to look through what systems might have issues:
Orchestration: the main agent decides to kick off an investigation
Subagents: it fans out investigation work to subagents which report on progress
Heuristics: the orchestrating agent can make speculative calls, then use heuristics to decide where to go deeper
There’s *a lot *of value in having an oncall engineer join the incident channel roughly five minutes after an alert fires, and to have a summary of investigations by AI agents. It gives the oncall dev a sense of what might be happening, instead of having to do all the “shovel work.”
**AI SREs typically interact with devs in Slack. **At startups that use Slack for comms, Slack is also the place to coordinate incidents; even pre-AI, it was common to open a dedicated Slack channel for an incident. Most AI SRE tools have a Slack bot in the incident channel; incident.io’s bot sits in the incident channel the whole time and takes steering instructions from engineers (look at the status of a specific system, ask it to dig up specific logs, etc.)
**AI SRE tools require a high level of trust, which is a reason to build in-house. **For these agents to be effective, they need access to telemetry, observability, logs, even the version control system because the more data they can access, the more effective they are. As Sam Newman writes in his book, ‘Building Resilient Distributed Systems’:
“For tools like incident.io’s AI SRE to work, they need access to information. This means that they will need broad access to a lot of valuable data if you want them to be effective. If the agent can’t see your telemetry data, then it can’t slice and dice that data for you. If it can’t look at your source code repository, then it can’t spot recent commits that may have caused the issue. This does mean that you need a high degree of trust in the vendor creating and running the agent.”
But how much do you trust a third-party vendor with all your data; potentially even a readonly version of your sanitized production database, which could come in handy for some investigations? The larger the company, the less likely that access to highly sensitive data will be granted. The trust issue is a major driver in why many mid-size-and-above companies build their own AI SRE tools.
Even if a vendor offers zero data retention (ZDR), there’s still a risk that its system could be compromised and leak information. Also, a vendor offering ZDR could be at a disadvantage when improving the system, compared to an in-house AI SRE tool that it’s possible to train on the data.
Our take is that AI SRE vendors are likely to become more capable than home-grown solutions because AI SRE startups obsess about better tooling fulltime; success depends on it. At larger companies, an AI SRE tool is more of a novel side project; engineers initially get excited but then move on to newer projects.
**Should AI SREs only mitigate, or also remediate? **Once the root cause analysis of an outage is done, and the AI SRE agent has helped to mitigate it, is the work complete? On one hand, the AI SRE tool typically also has access to the codebase, so should be able to come up with code-fix suggestions. On the other hand, pre-AI, you hardly ever saw an SRE team raise pull requests to fix the root causes of outages because they didn’t have the context or expertise for systems someone else built. But with AI, context can be passed on and expertise in writing code is pretty common for AI agents.
We asked the three AI SRE startups how they approach writing code to fix root causes, post-incident. They have different ways:
**Handing off PRs to coding agents.**incident.io is focused on passing incident context to engineers’ existing coding agents. The company built and demoed autonomous PR creation at its annual conference last year, but cofounder Chris Evans told us that customers generally prefer to use their own coding setups. The focus is therefore on giving agents the context they need to help engineers implement a fix.**A hybrid approach at ResolveAI.**The tool writes simple PRs itself and hands over larger changes to the customer’s coding agents.**Intend for agents to autonomously merge PRs at Rootly.**Their vision is to get to a point where their agent drafts a PR, and another agent judges whether it’s safe to merge, and when deemed safe, the merge happens automatically. If the merge is deemed risky, the oncall engineer still gets woken up. As yet, all PRs raised happen after a human has approved them.
All three vendors we talked to have rebuilt their investigation agent at least once because integrating AI into oncall looks simple, but that’s not the case. For example, ResolveAI built three different architectures in two years:
Scripted workflow. In 2024, models could not yet reliably chain tool calls. The startup built a pipeline of specific prompts like: “classify the alert“ → “gather logs+metrics“ → “extract insights” → “rank hypotheses.” This pipeline handled common incidents well, but did poorly when incidents did not “fit the script”.
Single agent with a knowledge graph. In 2025 when tool calling with AI agents became reliable, a primary agent took the alert → looked at a graph of services/dependencies/runbooks/deployments → decided what to do next. Overall, the accuracy improved but new problems appeared:
Context rot: for long, multi-service incidents, the agent ran into context saturation (the “dumb zone” as discussedon the Pragmatic Engineer podcast episodewith Dex Horthy). It started to request data it already had and to contradict its own conclusions.Self-confirmation: the agent was tasked to grade hypotheses it came up for likely causes of incidents. The agent almost always confirmed its first suspicion and didn’t examine its conclusion.Lack of ‘memory’: lessons from past investigations didn’t carry over to new ones.
Multi-agent system. In 2026, the team came up with three different agent roles:
Investigator: chases one hypothesis after anotherVerifier: challenges the conclusions of investigators. It’s structurally independent of investigators to avoid self-confirmationCommunicator: agent updates stakeholders
The Resolve team told us they find this system to be the most reliable and resilient in incident investigations.
At a high level, all three AI SRE startups follow similar steps for dealing with investigations:
Trigger: an alert fires or an engineer declares an incidentOrientation: pull in relevant data like telemetry and logs to find out what’s changed recently, like deployment and feature flagsFan out: kick off parallel investigative agents.Verify: challenge the conclusions of investigative agentsReport: update Slack and other communication channelsSteer: engineers give input to the agent, usually via Slack. This spawns more “fan out” investigationsHand off: an engineer or coding agent takes on suggested mitigation actions, like making a code change, restarting a system, etc.
There’s plenty of nuance in how to implement each step: our three vendors often do so in distinct ways! Here are tactics for each step:
Trigger: how do you categorize alerts: do you give fixed buckets for the agent (e.g. “needs human attention”, “can wait for business hours”, “noise”), or have the agent come up with a hypothesis for the cause before paging a human?Orient: do you measure the potential impact of the page (which can add latency), follow different rules based on the service impacted, or pull in a larger context graph?*Fan out:*do you fan out by hypothesis/potential failure type, or with as many agents as makes sense? What about long-running agents; do you wait for them, or limit tool-calling depth, or the cost of an investigation agent run?… and so on for other steps. You’ll want to decide on tradeoffs between latency, accuracy, and cost.
AI SRE tools need some kind of understanding of what the production system looks like: how systems are connected, the dependencies, and what caused things to go down previously…
