Measuring a security agent after it ships

After go-live, track six numbers: acceptance rate, revert rate, human minutes per accepted change, escalation rate, safeguard block rate and cost per task. Set each threshold from your own baseline before launch. Rival's agents normally saw about 5% of threads blocked by a model safeguard, and 19.2% on 5 Nov 2026. Read 19 Nov 2026.

Why measure outputs, not only uptime

The UK NCSC's secure AI guidelines say to measure "the outputs and performance of your model and system such that you can observe sudden and gradual changes in behaviour affecting security", and to treat "major updates like new versions" because changes to data, models or prompts change behavior. NIST AI RMF 1.0 control MANAGE 4.1 asks for "post-deployment AI system monitoring plans". Neither names a metric. The six below are this site's, chosen because each moves when a specific thing breaks.

The six metrics, thresholds and actions

Count a change as accepted only after the outcome is checked in the system, as in verify after each change. Thresholds are starting values chosen by this site, not published limits. Replace them with a band taken from the held-back set in the evals guide and your first four weeks.

Formulas and thresholds are this site's. The source for the reason to measure outputs is the NCSC guideline, read 19 Nov 2026.
MetricFormulaStarting thresholdAction when crossed
Acceptance rateaccepted proposals / proposals reviewedDown 10 points from baseline for 2 weeksRerun the held-back eval set; check whether inputs or the model changed
Revert rateaccepted changes later undone / accepted changesAbove 5% in any weekReview a sample of 59 accepted changes; stop widening what runs unattended
Human minutes per accepted changereview minutes / accepted changesDown 30% while revert rate is upReviewers are reading summaries: require the evidence panel to be opened
Escalation rateruns that hit a stop condition / runsDouble the baselineInspect the stop reasons; scope is too tight or inputs changed
Safeguard block ratethreads refused by the model provider / threadsDouble the baselineRetry, then switch to the fallback route; log which task types were blocked
Cost per taskmodel spend / tasks completed1.5 times baselineRead the cost guide; check retries and context growth

Each metric needs a baseline and an owner before go-live. The baseline is the median of the weekly values from the first four weeks after go-live in which the model, prompt and tools did not change, using only weeks with at least 100 reviewed proposals (or, for the block rate, at least 200 threads); a thinner week is pooled with the next one. Until four such weeks exist, use the held-back eval set result and mark the metric provisional.

Owners, windows and minimum counts are this site's starting values, to be replaced with your own.
MetricOwnerBaseline window and minimum count
Acceptance rateJob ownerMedian of 4 clean weeks, at least 100 proposals reviewed per week
Revert rateOwner of the system the change lands inMedian of 4 clean weeks, at least 100 accepted changes per week; a week with fewer is pooled
Human minutes per accepted changeLead reviewerMedian of 4 clean weeks, at least 100 accepted changes per week
Escalation rateEngineer who owns the harnessMedian of 4 clean weeks, at least 100 runs per week
Safeguard block rateEngineer who owns the harnessMedian daily rate over 4 clean weeks, at least 200 threads per day; low-volume days left out, as Rival does
Cost per taskEngineer who owns the harness, with the budget holderMedian of 4 clean weeks, at least 100 completed tasks per week

A four-week review worked through

The inputs below are invented for the example; the arithmetic is the point. An agent proposes dependency upgrades and a team reviews them weekly.

Invented inputs. Acceptance = accepted / reviewed; revert rate = reverted / accepted; minutes per accepted = review minutes / accepted.
WeekReviewedAcceptedRevertedReview minutesAcceptanceRevert rateMinutes per accepted
120014071,40070%5.0%10.0
218012661,26070%4.8%10.0
321014791,47070%6.1%10.0
41901141068460%8.8%6.0

Week 3 crosses the revert threshold (6.1% against 5%), so the team samples 59 of the 147 accepted changes. Week 4 crosses a second: minutes per accepted change fell 40%, from 10.0 to 6.0, while the revert rate rose to 8.8%. Faster review with more reverts is the signature of approval turning into a click. The action is to stop widening autonomy, re-read the 10 reverted changes for what the reviewer missed, and sample 59 of the 114 accepted. Acceptance fell from 70% to 60% in week 4, a drop of 10 points. The threshold needs two weeks, so it has not yet fired: it fires if week 5 is also at or below 60%. Until then the drop is a prompt to check whether the model or the inputs changed, and to rerun the eval set early if they did.

A real series: Rival's safeguard block rate

The safeguard block rate is the metric with a cause outside the team. Rival's post of 12 Nov 2026 says its package-analysis agents, which run on gpt-6.1-sol (a current model ID on OpenAI's model page, read 19 Nov 2026), "normally" see about one thread in twenty blocked by the model's cyber safeguard, and that the rate rose to one in five on 5 Nov. Its chart gives the share of threads blocked, by day the thread started, from production traces of 27 Oct to 9 Nov 2026 (low-volume days left out).

Rival Security (vendor report, 12 Nov 2026), share of agent threads that hit the cyber block on gpt-6.1-sol; read 19 Nov 2026. 30 and 31 Oct and 7 Nov are absent because Rival's chart leaves out low-volume days. The threshold column is this page's rule applied to Rival's figures.
Day (2026)Threads blockedAgainst a doubled 5% baseline (10%)
27 Oct5.8%Below
28 Oct5%Below
29 Oct4%Below
1 Nov2.9%Below
2 Nov4%Below
3 Nov7.6%Below
4 Nov6%Below
5 Nov19.2%Alert fires: first day above
6 Nov20.8%Above
8 Nov7.7%Below
9 Nov18.5%Alert fires again

Three of Rival's findings change how to read the metric, from its replays on 10 Nov 2026. Blocks vary by workload: across package-analysis threads of 27 Oct to 10 Nov, network-facing packages were blocked 16.1% of the time against 6.2% for the rest, so compare like with like. Blocks are not fixed: of 296 unchanged replays of 78 blocked requests, 115 (39%) were blocked again, 33 of the 78 were never blocked again, and with three retries 84% of blocked requests got through. And a block lands mid-reply, a median of 27 seconds in. So record blocks by task type, count retried successes separately, and keep a fallback route. These are Rival's measurements on one provider's model, not general rates.

The record kept at each review

AGENT REVIEW RECORD Agent and version: ____ (model, prompt, tool list versions) Review date: ____ Window: ____ to ____ Reviewer: ____ Proposals reviewed: ____ Accepted: ____ Reverted: ____ Review minutes: ____ Runs: ____ Escalations: ____ Threads: ____ Blocked: ____ Model spend: ____ Tasks completed: ____ Six metrics vs baseline: ____ Thresholds crossed: ____ Sample drawn (n and method): ____ Wrong items found: ____ Decision: continue / pause widening / rerun evals / roll back / switch route Owner of the action and date due: ____ Change in model, prompt or tools since last review: ____ (counts as a new version)

Download the template: .docx PDF CC BY 4.0

Name the owner and schedule before go-live. A weekly review suits most jobs. Keep the records with the run logs described in what to keep in an agent run log; the stop and rollback controls the decision depends on are on kill switch and rollback.

Sources

  1. UK NCSC, Guidelines for secure AI system development, secure operation and maintenance, read 19 Nov 2026
  2. NIST AI RMF 1.0 (NIST AI 100-1), MANAGE 4.1, read 19 Nov 2026
  3. Rival, When OpenAI's Cyber Classifier Flags Security Agents (12 Nov 2026), read 19 Nov 2026
  4. OpenAI, gpt-6.1-sol model page, read 19 Nov 2026

Last reviewed 19 Nov 2026