# Re-Evaluate After Model Changes: Questions Q16 and Q11

Re-run the same cases whenever the model, prompt, tools or data change, and on a timer in between. Anthropic's September 2025 postmortem shows why: bugs misrouted 0.8% of Sonnet 4 requests from 5 Aug and 16% in the worst hour on 31 Aug 2025, with no change on the customer side.

## The two questions to ask

- **Q16.** What re-runs the evaluation when the model version, prompt, tools or data change, and what result blocks the change from shipping?
- **Q11.** What do you measure after go-live to notice that the agent's behavior or quality has shifted, and who reviews it on what schedule?

Q16 is about changes you make. Q11 is about changes you did not make. The first needs a blocking rule, the second needs a clock. How to build the test set itself is on [AI agent evals](/guides/ai-agent-evals/) and [evaluate a security agent on your findings](/guides/evaluate-a-security-agent-on-your-findings/); this page assumes you have one.

## Where each requirement comes from

Quotations read on 19 Nov 2026.

| Question | Requirement as published | Source |
|---|---|---|
| Q16 | Update processes reflect "that changes to data, models or prompts can lead to changes in system behaviour (for example, you treat major updates like new versions)" | [NCSC, secure operation and maintenance](https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development/guidelines/secure-operation-maintenance) |
| Q16 | Practice C1: "Consider automating tests within a development pipeline as part of regression testing where possible." | [NIST SP 800-218A](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-218A.pdf) |
| Q16 | Regression evals "should have a nearly 100% pass rate" because a decline "signals that something is broken and needs to be improved" | [Anthropic, Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) |
| Q11 | "You measure the outputs and performance of your model and system such that you can observe sudden and gradual changes in behaviour affecting security." | NCSC, secure operation and maintenance |
| Q11 | MANAGE 4.1: "Post-deployment AI system monitoring plans are implemented, including mechanisms for capturing and evaluating input from users and other relevant AI actors, appeal and override, decommissioning, incident response, recovery, and change management." | [NIST AI RMF 1.0](https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf) |

## Which change re-runs what, and what blocks

The mapping is this site's; the principle that each change is treated like a new version is the NCSC's.

| Change | Re-run | Block shipping when |
|---|---|---|
| Model version or alias you set | Full set, 3 trials per case | A must-hold case fails any trial, or stable passes fall by more than the noise band |
| Prompt or task instructions edited | Full set, 3 trials | Same rule |
| Tool added, removed or parameters changed | Full set, plus the register review on [scope of change](/evaluation-checklist/scope-of-change/) | The tool list differs from the approved register |
| Data source or retrieval index changed | Cases that read that source | Answers change without a recorded reason |
| Supplier changes the model behind a name, or an alias moves | Full set within a time you wrote down | You cannot pin the old version (see [supplier controls](/evaluation-checklist/supplier-controls-when-buying/)) |
| Nothing you changed | A canary subset on a timer | Canary stable passes fall by more than the noise band |

The last row is the one teams omit. Anthropic's postmortem of Aug and Sep 2025 describes three infrastructure bugs that degraded Claude's responses without any change by customers. A context-window routing error began on 5 Aug 2025 and affected about 0.8% of Sonnet 4 requests; a load-balancing change on 29 Aug raised that, and in the worst hour, on 31 Aug, 16% of Sonnet 4 requests were affected. About 30% of Claude Code users who made requests in the period had at least one message routed to the wrong server type. A fix was deployed on 4 Sep. Anthropic wrote that early reports "were difficult to distinguish from normal variation in user feedback" ([Anthropic, 17 Sep 2025](https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues)). A fixed canary set run weekly gives a customer a comparison against its own baseline, which is what the early complaints lacked.

## Worked run: 48 cases, 3 trials

The figures below are invented to show the arithmetic. The formula comes from Anthropic: if an agent has a 75% per-trial success rate and you run 3 trials, "the probability of passing all three is (0.75)³ ≈ 42%", which is why one trial per case is not enough to see a change.

1. **Baseline.** A finding-triage agent has a regression set of 48 cases drawn from past findings, 12 of them marked must-hold (a wrong answer would close a real exposure). Each case runs 3 trials. On the current version, 45 cases pass all 3 trials, so the stable pass rate is 45/48 = 93.8%. Three cases fail and are known.
2. **Noise band.** Run the unchanged version two more times. The stable counts are 45, 44 and 45. The band is 1 case.
3. **Change.** The model alias is moved to a new version. The run gives 41 stable passes, 41/48 = 85.4%. Four cases that were stable now fail at least one trial, which is outside the band of 1.
4. **Rule.** Two of the four are must-hold cases. The release is blocked. The record names the cases, the trials that failed and the person who decides on a re-run or a rollback to the pinned version.
5. **Close.** The cause is found, a case that reproduces it is added to the set (now 49), and the next run on the new version gives 46 stable passes, 46/49 = 93.9%. Release is approved by the named role.

Anthropic's idea of graduating a capability case into the regression suite applies after launch: a task the agent passes reliably becomes a case that must keep passing.

## Q11: the clock after go-live

Ask for an owner, a schedule and a comparison point, in that order. A review with no owner is not a review. A schedule this site suggests, with the baseline kept from the last release run:

Cadences written for this page, not taken from a standard.

| Cadence | What is run or read | Compared against |
|---|---|---|
| Weekly | The 12 must-hold cases, 3 trials each, on the pinned model | The release baseline and the noise band |
| Weekly | Approval rate, revert rate and escalation rate (formulas on [measuring an agent after it ships](/guides/measuring-an-agent-after-it-ships/)) | The last four weeks, trended, not only an alarm line, because the NCSC asks for gradual change too |
| Monthly | 20 transcripts sampled at random, read by someone other than the builder | The same questions each month, noted in a short review record |
| On a supplier notice | The full set | The release baseline |

The review record is the evidence. Ask the owner for the last two review notes and what changed because of them.

## Test it on your own findings

1. Change something harmless in staging, such as one word in the prompt, and confirm the pipeline re-runs the set without anyone remembering to ask.
2. Break one must-hold case in the fixture on purpose and confirm the release is blocked and the named person is told.
3. Ask who gets the message when the supplier announces a model retirement, and how long the old version stays callable.

## Sources

- [NCSC, Guidelines for secure AI system development, secure operation and maintenance](https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development/guidelines/secure-operation-maintenance), read 19 Nov 2026
- [NIST SP 800-218A, practice C1](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-218A.pdf), read 19 Nov 2026
- [Anthropic, Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), read 19 Nov 2026
- [Anthropic, A postmortem of three recent issues (17 Sep 2025)](https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues), read 19 Nov 2026
- [NIST AI RMF 1.0 (AI 100-1), MANAGE 4.1](https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf), read 19 Nov 2026

Last reviewed 19 Nov 2026

Canonical: https://agenticaiincybersecurity.com/evaluation-checklist/re-evaluate-after-model-changes/
