Your AI Just Rewrote Its Own Playbook. How Do You Know It Got Better?

Press enter or click to view image in full size

There’s a pattern that shows up in every AI SRE and AI Ops conversation eventually. Someone runs a really impressive demo of an agent that handles production incidents. The agent diagnoses correctly. It escalates when it should. It recovers fast. Then someone in the back row asks the question that ends the meeting:

“But how do you know it’s actually getting better over time?”

Not “does it work today.” Getting better. Verifiably, measurably, with receipts. Before it’s been promoted to production.

Most vendors change the subject. We built the receipts.

The Flywheel Everybody Talks About

The Operational SRE/DBA Flywheel (blog post, doc) is a well-understood concept in theory: take real incidents, convert and couple them with the inject faults, diagnose, remediate, learn, repeat. Each incident makes the system smarter. The institutional knowledge grows. The library of known fixes compounds.

In practice, not everybody succeeds to close the loop. They have the injection. They have the agent. They don’t have the measurement that tells them whether the agent is actually getting smarter or just getting more confident while making the same mistakes. As stated earlier, consistency ≠ correctness.

This is the problem we tackle in this post. We specifically designed aiHelpDesk’s Vault around closing this loop with three distinct signals:

1. Consistency: does the agent reach the same conclusion under the same fault, reliably, across repeated runs? (STABLE/UNSTABLE cert)
2. Resolution Rate: does the playbook actually fix the problem?
3. Accuracy: is the agent’s diagnosis correct, independent of whether the fix worked by accident?

These three are not the same thing. A playbook with a 100% resolution rate can have a 40% accuracy rate if the remediation step happens to fix the problem regardless of what caused it. Conflating them is how you end up with a system that somehow “works”, but nobody can explain why and it’s unclear it will continue working for the next incident.

What We Actually Tested

Here’s what we ran.

If you want to reproduce the findings presented in this blog post, please feel free to check out this doc page for details on the commands to run and their output, but the excerpts presented here should be sufficient to make our point.

The fault is db-connection-refused (code link). PostgreSQL container stops, agent must diagnose and escalate to the SysAdmin Agent for container inspection (and possibly further to the K8s Agent if Postgres is hosted on K8s).

We started with v1.6, an AI-generated playbook update from vault suggest-update. Looked reasonable on paper. The guidance covered connection-refused, mentioned K8s pod inspection, had the right structure.

We ran it three times.

Stability report (3 runs):
Pass rate:    3/3 (100%)
Confidence:   min=57% max=97% range=40pp mean=80%
Verdict:      UNSTABLE

Three passes. One failure and… 40 percentage points of spread in how confident the agent was across identical faults. That’s not a stable playbook. That’s a coin flip that happens to land heads three times.

The judge scores told the same story: 33% / 67% / 100% (again, see this doc page for details). The agent was correct when it escalated, wrong when it didn’t and the playbook wasn’t giving it consistent enough guidance to know which situation it was in.

What Was Missing

One run out of three, the agent saw “connection refused” and got confused about whether to escalate or handle it directly. The v1.6 guidance didn’t clearly distinguish between three very different situations that all look like “database problem” from the outside:

• Connection refused: the PostgreSQL process is not running. Escalate to SysAdmin Agent.
• max_connections exhausted: the database IS running. Do NOT restart it. You’ll disconnect every active client for no benefit.
• pg_hba auth blocked: the database IS running. Do NOT restart it. Check the credentials.

Restarting a live database to “fix” a max_connections error is not a minor mistake. It’s 30 seconds of downtime plus a cascade of reconnect storms on top of an already-stressed system.

v1.7 named these failure modes explicitly.

• Step 1 became a classification step: read the exact error, identify which of four cases you’re in, then proceed accordingly.
• Step 2 added an explicit rule: if the hosting type is unknown, default
  to escalation — unknown infrastructure is itself a signal, not a reason to guess.

The Judge’s Preview

Before activating v1.7, we ran vault diff — judge:

  Verdict:            ✓  APPROVE
Guidance quality:   The new version significantly improves clarity by explicitly
distinguishing between connection-refused (process stopped) and
non-restart scenarios (max_connections, pg_hba auth, timeouts),
preventing harmful restarts.
Escalation safety:  Escalation safety is enhanced by adding a new criterion that
explicitly prevents restart when max_connections or pg_hba errors
occur (which indicate the database is running).

Haiku as the judge. Fast, cheap, independent of the agent model. The verdict is a prediction, stored with a timestamp. We’ll come back to this.

After Activation

We activated v1.7 and ran three more times.

  Stability report (3 runs):
Pass rate:    3/3 (100%)
Confidence:   min=95% max=100% range=5pp mean=97%
Verdict:      STABLE

5pp range. STABLE. Judge scores: 100% / 100% / 67%. The 67% on run 3 is worth naming: the agent escalated correctly in prose, but didn’t emit the exact ESCALATE_TO: pbs_sysadmin_docker_inspect structured token. That’s a real quality gap, not noise. Indeed, the playbook format specifies the token because the gate parser needs it. And that third run did not. So one out of three runs didn’t produce a machine-readable handoff signal.

The signal isn’t perfect. We’re showing you the score anyway. And that’s the point. We are not there yet, still need to iterate until we get to the correct and stable diagnosis.

The Metric Gap (And What We Found Inside It)

Here’s where it gets interesting. vault judge-accuracy cross-references judge predictions against actual outcomes. The first time we ran it:

  SERIES                     VERSION  JUDGE VERDICT   RUNS   SUCCESS%
───────────────────────────────────────────────────────────────────
pbs_db_restart_triage      1.7      APPROVE           3     0%
pbs_db_restart_triage      1.6      APPROVE           3     0%

Huh? APPROVE → 0% SUCCESS. For both versions. Which looks like the judge is useless because it approves promoting a new playbook version… which doesn’t lead to a success in resolving incidents?

That’s not it and the judge wasn’t useless. We pulled the actual run records from the database:

  plr_936512c6 | pb_2c498091 | gate_pending | pbs_sysadmin_docker_inspect
plr_61b1aaae | pb_2c498091 | gate_pending | pbs_sysadmin_docker_inspect
plr_807f0ae5 | pb_2c498091 | gate_pending | pbs_sysadmin_docker_inspect

gate_pending. Ah-ha. The escalation gate did fire. And the agent had correctly emitted ESCALATE_TO: pbs_sysadmin_docker_inspect. The gateway intercepted it, recorded the handoff target and was waiting for the SysAdmin chain to complete. In our repeat-mode test runs, the SysAdmin stage never runs and so the metric saw zero resolved incidents and called it 0%.

That’s wrong. This is a category error baked into the metric: resolved means the full chain completed. For a triage-only playbook whose job is to hand off correctly, “correctly escalated” is the success criterion, not “resolved end-to-end by a single agent.”

A single SUCCESS% column hides this distinction. We fixed it:

  SERIES                     VERSION  JUDGE VERDICT   RUNS   RESOLVED%   ESCALATED%
───────────────────────────────────────────────────────────────────────────────────
pbs_db_restart_triage      1.7      APPROVE           3     0%          100%
pbs_db_restart_triage      1.6      APPROVE           3     0%          66%

Now the story is checks out.

• v1.6: APPROVE, 66% escalated.
• v1.7: APPROVE, 100% escalated.

The 66% matches exactly with one failed run in the v1.6 UNSTABLE cert. That’s the same run that didn’t escalate is the run where escalated_to is empty in the DB. Three independent signals. One story.

Why are we bringing this up? Because this is what the accountability loop is supposed to do: surface when a metric is lying, not hide it.

The Bill of Rights Connection

What we built today directly implements two rights from the aiHelpDesk Customer Bill of Rights.

Right II: The Right to a Second Opinion. The LLM judge is an independent check on playbook improvement proposals before they reach you. The judge doesn’t share a context window with the agent. It doesn’t know which version is “supposed” to be better. It reads the diff and tells you whether the knowledge change is an improvement or a regression. When the prediction tracks reality (APPROVE → 100% ESCALATED), you have evidence the judge is calibrated, not just opinionated.

Right IV: The Right to Know the Grade. vault judge-accuracy, vault calibration, vault versions: these exist so you can verify claims, not just trust them. The calibration Data Quality Banner activates automatically when human feedback is sparse and the grade is driven by LLM self-assessment instead of human verdicts. The system tells you when not to trust its own numbers.

The Bill of Rights is explicit about gaps. Right IV currently has one: calibration data is blended across model versions until aiHelpDesk release v0.20 ships the model-scoped cert PK. Publishing the gap is the commitment. A vendor who claims all ten rights without qualification is making claims they can’t verify.

The Deeper Point

Informed Consent has three parts: Informed, Consent, Right to Refuse. The first part — being informed of the diagnosis — is only meaningful if the diagnosis is actually correct. “We showed you what the AI said” is not the same as “we showed you something true.”

The flywheel closes this gap: vault accuracy tracks diagnosis correctness. vault calibration checks whether confidence predicts correctness. vault judge-accuracy checks whether the judge’s improvement predictions track real outcomes. Together, they turn Informed Consent from a process claim into a measured property.

Without the measurement loop, you’re consenting to a diagnosis whose correctness nobody is tracking. That’s not consent. That’s deference.

What This Looks Like at 2am

Your monitoring alert fires. db-connection-refused (code link). The triage agent runs, emits ESCALATE_TO: pbs_sysadmin_docker_inspect and hits the escalation gate. You get a notification.

At the gate: you see the agent’s hypothesis, the confidence level, the evidence it cited and the handoff target. You know this playbook has a STABLE cert with 5pp variance and 100% correct escalation rate. You know the judge APPROVEd the last update and that prediction held.

You approve. The SysAdmin Agent inspects the container. The database comes back up.

Three weeks later, an incident causes a new playbook draft to be proposed. You run vault diff — judge. If it says APPROVE and the reasoning is sound, you activate. If it says NEEDS_REVIEW, you read why before deciding. If subsequent runs show APPROVE → 40% ESCALATED, you know the judge was wrong and you tune the rubric.

This is the loop. The loop has receipts.