Verify after each change an agent makes

Verifying an agent's work means checking the changed system, not the agent's report. Question Q10 asks for a separate check after every change: a query, a test or a diff against the real system. This page gives the checks by change type, how each fails and a worked lockfile case on CVE-2026-75604.

Why the agent's report is not the evidence

Anthropic, Demystifying evals for AI agents (published 9 Jan 2026): "A flight-booking agent might say 'Your flight has been booked' at the end of the transcript, but the outcome is whether a reservation exists in the environment's SQL database." In Building effective agents the same company says agents need "ground truth" from the environment at each step. A plausible report is cheap. An agent can say a package was upgraded or a port closed with none of it true, so the check is the only record that the world changed.

Worked example: a dependency upgrade for CVE-2026-75604

The CVE facts are from the advisory records read on 19 Nov 2026; the repository, versions in use and counts are invented for the example. CVE-2026-75604 affects Next.js applications that use the Pages router, or the App router without Cache Components, when the server is hosted on machines using a Windows filesystem. Backslashes in route segments are not consistently escaped before incremental-cache paths are built, so a remote request can traverse outside the cache root and read private build data. Disclosure of the server-reference-manifest encryption key can enable remote code execution, which depends on the key being usable against server actions. The GitHub advisory GHSA-p293-qw3h-jr36 was published on 25 Aug 2026; NVD published the CVE record on 1 Sep 2026 with CVSS 3.1 base score 9.0 (supplied by GitHub). The advisory gives no known workaround. Affected: next 13.4.0 up to but excluding 15.5.24, and 16.0.0 up to but excluding 16.3.3. Fixed: 15.5.24 and 16.3.3.

A team's agent reports: "Upgraded next to 15.5.24 on the web app, tests pass, ready to merge." The web app is deployed on Windows servers with the Pages router, so it is in scope. The harness then runs the checks the agent cannot skip.

Illustrative run. Versions and counts are invented; the affected and fixed versions are from the advisory.
CheckRun byResult for the agent's claimWhat it proves
Diff touches only package.json and the lockfileCIPass: 2 files, 14 linesNo other code was changed
Resolved version in the lockfile is outside the affected range (read with npm ls next)Script, read-onlyFail: resolves to 15.5.22, below 15.5.24The manifest says 15.5.24 but the install does not
Full test suite on the branchCINot reached until the lockfile is regeneratedTests passing on a stale lockfile prove nothing about the fix
Version on the deployed serverPost-deploy jobRun after merge: must print 15.5.24 or higherThe running system matches the repository

The second check is the one that matters. The agent's report was wrong in a way no one would see from the pull request title. The harness opens a ticket, sends the branch back, and counts one mismatch against the agent's record. After the fix the post-deploy job prints the version, and the ticket closes only then. The checks that ask the real system are what detect a wrong report; tests alone would have passed on the stale lockfile.

Checks by kind of change, how each can fail, and the record it leaves

Illustrative checks, written for this page.
Change the agent reportsSeparate check against the real systemHow the check itself failsRecord the check leaves
Public access to a storage bucket blockedRead the bucket configuration from the provider API after the call, with a read-only roleThe role reads cached credentials or a different account or region than the one changed, so it reports the old state or the wrong bucketAPI response with account, region, bucket and read time, stored beside the change entry
Ticket created and assignedQuery the ticketing system for the ID, assignee and queueThe check role cannot see the queue and reports "not found", or it matches on title and finds an older ticketTicket ID, queue and assignee as read at check time
Package upgraded and deployedPost-deploy job prints the installed version on each running hostThe job reads the staging host, or one pool member that already had the new version while others still run the old oneHost names and versions printed, with the deploy ID
Finding closed as not exploitableA person samples the reasoning against the code path on a scheduleThe reviewer reads only the agent's summary, not the code pathReviewer, sample size and disagreements
Firewall or policy change draftedPolicy simulation or review tool shows the effect before applyThe simulation runs against a stale copy of the configurationSimulation output with the hash of the configuration it used

What a good answer looks like

  • Every write tool in the register on scope of change has a named check that queries the system the tool changed.
  • The check runs outside the agent's control, and the agent cannot write to the thing the check reads.
  • A failed check opens a ticket and can trigger the stop path on stop and handover conditions.

Test it on your own systems

  1. Take ten recent agent changes, run the named check now, and count mismatches.
  2. In staging, make a tool return success without changing anything and confirm the check catches it.
  3. Break the check, not the tool: point it at the wrong host or an expired role and confirm it reports an error instead of a pass. Compare your findings with the offline version of the same test in evaluate a security agent on your findings.

Sources

  1. Anthropic, Demystifying evals for AI agents (published 9 Jan 2026), read 19 Nov 2026
  2. Anthropic, Building effective agents, read 19 Nov 2026
  3. Next.js security advisory GHSA-p293-qw3h-jr36 (CVE-2026-75604), read 19 Nov 2026
  4. Next.js 15.5.24 release notes, read 19 Nov 2026
  5. NVD, CVE-2026-75604 record, read 19 Nov 2026

Last reviewed 19 Nov 2026