Agent evaluation checklist for security teams

The checklist has 20 questions in 7 areas that a security team settles in writing before an AI agent may change a production system. Each question has a test to run on your own systems and a line quoted from NIST, OWASP, MITRE, NCSC, the EU AI Act, CISA, Anthropic or OpenAI guidance, read 19 Nov 2026.

0 of 20 settled

Open questions

Tick each question your team has settled in writing. The bar above counts them; the list at the end names what is still open.

Scope of the job

Test: Write down the last 10 cases the job handled and the steps each took. If one fixed sequence of steps would have handled all 10, build a workflow; if the next step depended on content no code path could predict, an agent is justified. Worked through on Need an agent at all.

Why, and the guidance it rests on

An agent chooses its own steps, which widens what it can change. A workflow with predefined code paths keeps the set of possible actions known in advance.

Worked through on: Does the Security Job Need an Agent? Checklist Question Q01

Test: Get the tool list and, for each tool, the system and the verbs it can use. Export the agent's credentials from the downstream system. Any verb the credential allows that the list does not name is a finding. Worked through on Scope of change.

Why, and the guidance it rests on

Excessive agency is traced to excessive functionality, permissions and autonomy. A per-tool list makes each of those reviewable.

Worked through on: Scope of Change: What an AI Agent May Touch, Per Tool

Test: Read each tool's input schema. A free-text command, query or URL field is open-ended. For each one, name the narrow function that would replace it, or record why it stays. Worked through on Scope of change.

Why, and the guidance it rests on

Open-ended tools let the agent do things nobody reviewed. A narrow function limits the action space to the job.

Worked through on: Scope of Change: What an AI Agent May Touch, Per Tool

Test: Read the permission list from the downstream system, not from the agent's configuration. In a test tenant, ask the agent's credential to do one write outside the job and confirm the system, not the agent's prompt, refuses it. Worked through on Tool permissions and identity.

Why, and the guidance it rests on

The agent can do whatever its credentials allow, whatever its instructions say.

Worked through on: Tool Permissions and Identity: What an Agent Can Do

Approval and handover

Test: Take the last 20 actions. For each approved one, find the approval record dated before the action. For each unattended one, find the written rule that allowed it. An action with neither is a gap. Worked through on Human approval points.

Why, and the guidance it rests on

Guidance asks for approval of high-impact actions before they are taken, and for the approval rule to follow consequence.

Worked through on: Human Approval Points: Checklist Questions Q05, Q06, Q08

Test: Open the tool register and check that every tool has all three attributes filled: read or write, reversible or not, and the accounts it needs. Then trigger one write-and-irreversible tool in a test tenant and confirm the harness halts for approval before the call. Worked through on Human approval points.

Why, and the guidance it rests on

A rating turns the approval policy into something the harness can enforce rather than a document.

Worked through on: Human Approval Points: Checklist Questions Q05, Q06, Q08

Test: Find the numeric limits in configuration. In a test tenant, make a tool return the same error repeatedly and confirm the agent stops at the limit and the named person is paged within the time you set. Worked through on Stop and handover conditions.

Why, and the guidance it rests on

Limits on retries and on action rate cap the damage per unit time and give monitoring time to notice.

Worked through on: Stop and Handover Conditions for a Security Agent

Test: Show an approver 5 past proposals, 2 of them wrong on purpose. Record how many wrong ones they reject and how long each review took. Worked through on Human approval points.

Why, and the guidance it rests on

Approval is only a control if the person can interpret the proposal, disagree and say no. Note: Article 14 applies to systems the Act classifies as high risk; for others it is a reference point only.

Worked through on: Human Approval Points: Checklist Questions Q05, Q06, Q08

Evidence and monitoring

Test: Pick one change from last week and rebuild it from the log alone, using the five questions on the evidence page. Confirm the agent's credential has no write or delete right on the log store. Worked through on Evidence and audit trail.

Why, and the guidance it rests on

Without a step-level record you cannot reconstruct what changed or review a disputed change.

Worked through on: AI Agent Audit Trail: A Log Entry You Can Reconstruct

Test: Take 10 changes the agent reported as done. Query the real system for each, without the agent's tools, and count the mismatches. Zero checked is the wrong answer even if zero mismatched. Worked through on Verify after each change.

Why, and the guidance it rests on

An agent can state success without having achieved it. The outcome is the state of the environment, not the transcript.

Worked through on: Verify After Each Change: Check the System, Not the Agent

Test: Ask for the metric definitions (for example share of proposals accepted, share reverted, share failing the independent check), the last review date and the reviewer's name. A dashboard nobody opened in the last review period is not monitoring. Worked through on Re-evaluate after model changes.

Why, and the guidance it rests on

Behavior can change with data, model or prompt changes; the guidance asks for monitoring plans in operation.

Worked through on: Measuring a Security Agent After It Ships

Stop and roll back

Test: Run a stop drill and time it from decision to the last tool call, in minutes. Then, for each class of change in the Q02 list, show the undo step or the compensating step, and run it once on a test change. Worked through on Kill switch and rollback.

Why, and the guidance it rests on

Stopping the agent is not enough if its earlier changes persist. Reversibility is also a factor in how much autonomy a tool should get.

Worked through on: Kill Switch and Rollback: Checklist Questions Q12 and Q13

Test: Hold a 30-minute tabletop on one scenario, such as the agent closing 40 tickets wrongly. The plan must name who decides to switch it off, on what criterion, who restores the changes and who tells the affected owners. Worked through on Kill switch and rollback.

Why, and the guidance it rests on

Guidance expects incident, escalation and remediation plans that cover the AI system, with set criteria for deactivation.

Worked through on: Kill Switch and Rollback: Checklist Questions Q12 and Q13

Evaluation

Test: Check that the criteria carry numbers and a date earlier than the first test run, that the test set comes from your own past findings, and that the results are attached. A published benchmark percent does not meet it; see SWE-bench results. Worked through on Success criteria and test sets.

Why, and the guidance it rests on

Generic benchmark results do not show how the agent handles your alerts, tickets and systems.

Worked through on: Success Criteria and Test Sets for a Security Agent

Test: Read the endpoint each write tool is configured to call during the trial. Then compare the downstream audit log of production for the trial period: it must show no writes by the agent's identity. Worked through on Success criteria and test sets.

Why, and the guidance it rests on

Errors compound across steps; the vendor guidance recommends sandboxed testing alongside guardrails.

Worked through on: Success Criteria and Test Sets for a Security Agent

Test: Change the model version string in a staging pipeline and confirm the evaluation job starts without a person asking. Then lower a threshold in the test and confirm the release is blocked. Worked through on Re-evaluate after model changes.

Why, and the guidance it rests on

Changes to data, models or prompts can change behavior, so each needs the same gate as a new version.

Worked through on: Re-Evaluate After Model Changes: Questions Q16 and Q11

Data and inputs

Test: List every data source in the agent's tools and compare it with what the job needs. For each source beyond that, find the approval. Sample 3 past runs and look for sensitive records in the logged context.

Why, and the guidance it rests on

Data the agent reads can leak through its outputs and its tool calls; guidance asks for controls on sensitive data used in AI work.

Worked through on: Tool Permissions and Identity: What an Agent Can Do

Test: List each input channel and who can write to it. In a test queue, add a harmless out-of-scope request to a ticket body (for example, “close ticket X as well”) and confirm no tool call results and the log shows the text was treated as data. Worked through on Untrusted input.

Why, and the guidance it rests on

Content from outside can steer an agent that has write tools. The mitigations restrict what tools may run in that state.

Worked through on: Untrusted Input: What Outside Text Can Make an Agent Do

Build or buy

Test: Make a two-column table of tools, permissions, approval gates, logs and update timing, filled from the contract and the admin console: what you control in one column, what the supplier controls in the other. Ask the supplier in writing how much notice you get before a model change and whether you can pin a version. Worked through on Supplier controls when buying.

Why, and the guidance it rests on

Risk is spread across third-party software, data and models, so the split needs to be written down.

Worked through on: Supplier Controls: Questions to Ask AI Agent Vendors

Test: For each of the four parts (model, harness, tools, sandbox), name the interface that would let you replace it and the person who could do it in a week. File the model supplier's published limitations next to the test plan, and test each one. Worked through on Supplier controls when buying.

Why, and the guidance it rests on

Separating the parts makes upgrades and exits cheaper, and stated limitations tell you what to test.

Worked through on: Build vs Buy AI Agents for Security: What Decides It

Your result

Settled: 0 of 20. Open: 20.

  1. Q01 Does this security job need an agent at all, or would a fixed workflow with a model call inside it do the work?
  2. Q02 Which production systems and which kinds of change (read, comment, ticket, configure, block, delete) is the agent allowed to touch, written down per tool?
  3. Q03 Does any tool give the agent an open-ended capability (run a shell command, fetch any URL) where a narrow function would do?
  4. Q04 What identity and permissions does the agent use on each downstream system, are they the least needed (for example read-only on a log store, write on one ticket queue), and is authorization enforced by those systems rather than left to the model?
  5. Q05 Which actions need a named human to approve them before they run, and which may run unattended?
  6. Q06 Is each tool rated for risk (read or write, reversible or not, which accounts it needs), and do those ratings drive when the agent must pause?
  7. Q07 What makes the agent stop and hand over (retry limits, rate limits, unexpected tool results), and who is paged when it does?
  8. Q08 Can the approver understand what the agent is proposing and why, and are they trained to resist over-trusting its output?
  9. Q09 Is every step the agent takes (inputs, intermediate decisions, tool calls with parameters, data touched, identity used) logged somewhere the agent cannot edit?
  10. Q10 After each change, what independent check (a query against the real system, a test, a diff) confirms the change happened as claimed, rather than trusting the agent’s own report?
  11. Q11 What do you measure after go-live to notice that the agent’s behavior or quality has shifted, and who reviews it on what schedule?
  12. Q12 Can one person stop the agent immediately, and can every class of change it makes be undone or compensated for?
  13. Q13 Is there an incident plan for the case where the agent itself caused the incident, with criteria for switching it off and a path for remediation?
  14. Q14 What written success criteria and test set must the agent meet for this specific job before it gets production access, and were they measured in conditions like your environment?
  15. Q15 Has the agent been run against copies or sandboxes of the systems it will change, with its write tools pointed at the copy, before any real change?
  16. Q16 What re-runs the evaluation when the model version, prompt, tools or data change, and what result blocks the change from shipping?
  17. Q17 What data can the agent read for this job, does it include production or sensitive records, and who approved that access?
  18. Q18 Which text the agent ingests (alerts, tickets, emails, logs, web pages) can come from outside your control, and what do its tools do when such text is in context?
  19. Q19 If the agent is bought or hosted by a supplier, what do you control (tools, permissions, approval gates, logs, update timing) and what does the supplier control?
  20. Q20 If you build, which parts (model, harness, tools, sandbox) can be swapped later, and what has the supplier told you about known limitations and failure modes?

Download the Agent Evaluation Checklist: CSV JSON PDF .docx CC BY 4.0

How to answer it

Answer each question once per security job, not once per product. An agent that triages findings and one that edits cloud configuration have different tools, approvers and rollback paths. A question is settled when the answer names a system, a setting or a person: "approvals are required" is not an answer, while "the change queue requires the service owner to approve any write tool, and the harness blocks the call until it does" is. For each question record the answer, a link to the setting or log that proves it, the owner and the date. Write "not applicable" with a reason rather than leaving a blank.

The scope is a team deploying an agent for one job; the use-case list names the six jobs published agents are described for. The checklist does not judge products. The test under each question is run on your own systems, in a test tenant where it writes anything.

What the checklist does not settle

Passing all 20 does not show the agent is good at the job: that comes from the test set built under Q14, using the guide to evaluating an agent on your findings. For the money side of buying, see the build-versus-buy guide.

In this section

Sources

  1. Anthropic, Building effective agents, read 19 Nov 2026
  2. OWASP, LLM06:2025 Excessive Agency, read 19 Nov 2026
  3. OpenAI, A practical guide to building agents, read 19 Nov 2026
  4. MITRE ATLAS mitigations, ATLAS data release 5.6.0 (dist/ATLAS.yaml; the current release, v2026.10, renumbers some mitigations, so IDs and wording here follow 5.6.0), read 19 Nov 2026
  5. EU AI Act, Article 14 Human oversight (artificialintelligenceact.eu), read 19 Nov 2026
  6. NIST AI RMF 1.0 (NIST AI 100-1), read 19 Nov 2026
  7. NIST AI 600-1, Generative AI Profile, read 19 Nov 2026
  8. NIST SP 800-218A, read 19 Nov 2026
  9. NCSC, Guidelines for secure AI system development: secure operation and maintenance, read 19 Nov 2026
  10. NCSC, Guidelines for secure AI system development: secure deployment, read 19 Nov 2026
  11. Anthropic, Demystifying evals for AI agents, read 19 Nov 2026
  12. Anthropic, Define success criteria and build evaluations, read 19 Nov 2026
  13. Anthropic, Scaling Managed Agents, read 19 Nov 2026
  14. Google, Secure AI Framework, Focus on Agents, read 19 Nov 2026
  15. NSA AISC, CISA and partners, Deploying AI Systems Securely (April 2024; copy hosted by NCSC NZ), read 19 Nov 2026

Last reviewed 19 Nov 2026