Agentic On-Call: agents that show their work
How my team built investigation agents that trace every step, report what they don't know, and are tested against past incidents with known answers.
Situation
When an alert fires, the on-call engineer often can't get to a root cause without the one expert who has seen the problem before. That expert is asleep, busy, or on another team. The expertise exists. It just isn't reachable at 3am.
Constraint
Agents can already produce a plausible answer. On-call engineers won't act on one they can't check. An answer that's hard to inspect is also hard to improve, and hard to reuse on the next incident.
What I did
I shaped the architecture with the team around four decisions.
- Reusable units of expertise. Each team encodes how its experts investigate, as building blocks other teams can share across domains.
- Every step traced. An investigation records each step it took and the evidence it used, so an engineer can check the path, not just the conclusion.
- Gaps reported, not guessed. When the agent lacks the expertise for a problem, it says so. That report tells us what expertise to capture next.
- Regression tests from past problems. Golden datasets of incidents we've already solved, with known answers, so a change that makes the agent worse shows up before an on-call engineer finds it.
We proved it on our own on-call first.
What didn't work
Asking other teams to learn the authoring model and encode their own expertise produced interest but little follow-through, because they had their own commitments. So we changed the model. Our engineers captured how each team's experts investigate, and the experts reviewed and corrected the result. That made the value concrete without asking for their time up front.
Result
Engineering teams adopted it within two months. We demoed it at Splunk .conf26, where seven enterprises, including major financial institutions, asked to become design partners.