Verify after each change an agent makes
Verifying an agent's work means checking the changed system, not the agent's report. Question Q10 asks for a separate check after every change: a query, a test or a diff against the real system. This page gives the checks by change type, how each fails and a worked lockfile case on CVE-2026-75604.
Why the agent's report is not the evidence
Anthropic, Demystifying evals for AI agents (published 9 Jan 2026): "A flight-booking agent might say 'Your flight has been booked' at the end of the transcript, but the outcome is whether a reservation exists in the environment's SQL database." In Building effective agents the same company says agents need "ground truth" from the environment at each step. A plausible report is cheap. An agent can say a package was upgraded or a port closed with none of it true, so the check is the only record that the world changed.
Worked example: a dependency upgrade for CVE-2026-75604
The CVE facts are from the advisory records read on 19 Nov 2026; the repository, versions in use and counts are invented for the example. CVE-2026-75604 affects Next.js applications that use the Pages router, or the App router without Cache Components, when the server is hosted on machines using a Windows filesystem. Backslashes in route segments are not consistently escaped before incremental-cache paths are built, so a remote request can traverse outside the cache root and read private build data. Disclosure of the server-reference-manifest encryption key can enable remote code execution, which depends on the key being usable against server actions. The GitHub advisory GHSA-p293-qw3h-jr36 was published on 25 Aug 2026; NVD published the CVE record on 1 Sep 2026 with CVSS 3.1 base score 9.0 (supplied by GitHub). The advisory gives no known workaround. Affected: next 13.4.0 up to but excluding 15.5.24, and 16.0.0 up to but excluding 16.3.3. Fixed: 15.5.24 and 16.3.3.
A team's agent reports: "Upgraded next to 15.5.24 on the web app, tests pass, ready to merge." The web app is deployed on Windows servers with the Pages router, so it is in scope. The harness then runs the checks the agent cannot skip.
| Check | Run by | Result for the agent's claim | What it proves |
|---|---|---|---|
| Diff touches only package.json and the lockfile | CI | Pass: 2 files, 14 lines | No other code was changed |
| Resolved version in the lockfile is outside the affected range (read with npm ls next) | Script, read-only | Fail: resolves to 15.5.22, below 15.5.24 | The manifest says 15.5.24 but the install does not |
| Full test suite on the branch | CI | Not reached until the lockfile is regenerated | Tests passing on a stale lockfile prove nothing about the fix |
| Version on the deployed server | Post-deploy job | Run after merge: must print 15.5.24 or higher | The running system matches the repository |
The second check is the one that matters. The agent's report was wrong in a way no one would see from the pull request title. The harness opens a ticket, sends the branch back, and counts one mismatch against the agent's record. After the fix the post-deploy job prints the version, and the ticket closes only then. The checks that ask the real system are what detect a wrong report; tests alone would have passed on the stale lockfile.
Checks by kind of change, how each can fail, and the record it leaves
| Change the agent reports | Separate check against the real system | How the check itself fails | Record the check leaves |
|---|---|---|---|
| Public access to a storage bucket blocked | Read the bucket configuration from the provider API after the call, with a read-only role | The role reads cached credentials or a different account or region than the one changed, so it reports the old state or the wrong bucket | API response with account, region, bucket and read time, stored beside the change entry |
| Ticket created and assigned | Query the ticketing system for the ID, assignee and queue | The check role cannot see the queue and reports "not found", or it matches on title and finds an older ticket | Ticket ID, queue and assignee as read at check time |
| Package upgraded and deployed | Post-deploy job prints the installed version on each running host | The job reads the staging host, or one pool member that already had the new version while others still run the old one | Host names and versions printed, with the deploy ID |
| Finding closed as not exploitable | A person samples the reasoning against the code path on a schedule | The reviewer reads only the agent's summary, not the code path | Reviewer, sample size and disagreements |
| Firewall or policy change drafted | Policy simulation or review tool shows the effect before apply | The simulation runs against a stale copy of the configuration | Simulation output with the hash of the configuration it used |
What a good answer looks like
- Every write tool in the register on scope of change has a named check that queries the system the tool changed.
- The check runs outside the agent's control, and the agent cannot write to the thing the check reads.
- A failed check opens a ticket and can trigger the stop path on stop and handover conditions.
Test it on your own systems
- Take ten recent agent changes, run the named check now, and count mismatches.
- In staging, make a tool return success without changing anything and confirm the check catches it.
- Break the check, not the tool: point it at the wrong host or an expired role and confirm it reports an error instead of a pass. Compare your findings with the offline version of the same test in evaluate a security agent on your findings.
Sources
- Anthropic, Demystifying evals for AI agents (published 9 Jan 2026), read 19 Nov 2026
- Anthropic, Building effective agents, read 19 Nov 2026
- Next.js security advisory GHSA-p293-qw3h-jr36 (CVE-2026-75604), read 19 Nov 2026
- Next.js 15.5.24 release notes, read 19 Nov 2026
- NVD, CVE-2026-75604 record, read 19 Nov 2026
Last reviewed 19 Nov 2026