Where a security agent must stop and ask a person
Ask for a written approval map: every action the agent can take, set to unattended, approve first or never, with a named approver role. Anthropic reported on 25 Mar 2026 that Claude Code users accept 93% of permission prompts, so test whether your approvals are decisions. Sources read 19 Nov 2026.
The three questions to put to a vendor or a builder
Checklist group "Human approval points" holds three questions. Ask them in this order, because each depends on the one before.
- Q05. Which actions need a named human to approve them before they run, and which may run unattended?
- Q06. Is each tool rated for risk (read or write, reversible or not, which accounts it needs), and do those ratings drive when the agent must pause?
- Q08. Can the approver understand what the agent is proposing and why, and are they trained to resist over-trusting its output?
This page covers where the pause goes and whether the pause works. What the approver should see on the screen (diff, reason, evidence, undo step) is on the human approval design guide, and the spectrum from approving every step to watching a dashboard is on the autonomy levels guide.
Where each requirement comes from
| Question | Requirement as published | Source |
|---|---|---|
| Q05 | "Utilise human-in-the-loop control to require a human to approve high-impact actions before they are taken." | OWASP LLM06:2025 Excessive Agency |
| Q05 | Approval policies "should follow the degree of consequence of the task at hand"; the approver "may be technical staff or business unit SMEs depending on the use case"; dedicated audit agents "may assist human approval, but final adjudication should be conducted by a human decision-maker" | MITRE ATLAS mitigation AML.M0029, data release 5.6.0 |
| Q06 | Rate each tool "low, medium, or high" using "read-only vs. write access, reversibility, required account permissions, and financial impact", then "use these risk ratings to trigger automated actions, such as pausing for guardrail checks before executing high-risk functions or escalating to a human if needed" | OpenAI, A practical guide to building agents, tool safeguards |
| Q06 | "Actions that are sensitive, irreversible, or have high stakes should trigger human oversight until confidence in the agent's reliability grows." | Same OpenAI guide, high-risk actions |
| Q08 | The overseer is enabled "to remain aware of the possible tendency of automatically relying or over-relying on the output" and "to decide, in any particular situation, not to use the high-risk AI system" | EU AI Act, Article 14(4) |
What a good answer contains
The answer to Q05 is a document, not a sentence. Ask to see it. This is the template, with one row filled in from a published incident-log example.
APPROVAL MAP for
| Action class | Tool | Risk (low/medium/high) | State (unattended / approve first / never) | Approver role | If nobody answers in |
Evidence shown with the proposal |
|---|---|---|---|---|---|---|
| Read a finding | read_finding | Low | Unattended | none | n/a | n/a |
| Comment on a ticket | add_ticket_comment | Low | Unattended, logged | none | n/a | n/a |
| Open a pull request | open_pull_request | Medium | Approve first (merge) | Code owner of the repository | Reject | Diff, test result, rescan result |
| Change a cloud setting | apply_config_change | High | Approve first, before each call | Cloud platform on-call | Reject | Current value, new value, prior-state copy, owner of the asset |
| Delete a remote branch or a resource | delete_resource | High | Never | n/a | n/a | n/a |
Rules: the state is stored in the harness configuration and enforced there. A tool missing from this table is not callable. Changing a state needs a ticket signed by
Three checks make the answer good, and each can be inspected in minutes.
- Q05. Every action has one of three states and "approve first" names a role. "The team" and "a human" fail.
- Q06. The risk label and the pause rule live in the harness configuration. If you change a label in the file and the agent's behavior does not change, the label is decoration.
- Q08. The approver can say no and has done so. A refusal rate of zero over months is a finding.
Worked example: sorting a request into the three states
Anthropic's account of how it built Claude Code auto mode lists real cases from its internal incident log. One is a user who asked the agent to "clean up old branches"; the agent listed remote branches, built a pattern match and issued a delete. Anthropic's reasoning for blocking it: "the request was vague, the action irreversible and destructive, and the user may have only meant to delete local branches" (Anthropic, 25 Mar 2026).
Run that case through the template above. The tool deletes remote state (write), cannot be undone from the agent's side (irreversible) and needs a repository admin credential (account permissions). OpenAI's guide lists four factors for rating a tool: read-only versus write access, reversibility, required account permissions and financial impact. The case gives no cost figure, so three of the four apply, and all three point to high. The state is "never" for the agent, and a person runs the deletion from the repository's own console after reading the list. A different request, "summarize the open findings on repo X", uses only read_finding and stays unattended. The two requests sound alike to a user and land in opposite states, which is why the map is per tool and not per conversation.
Test it on your own findings
The approval step fails quietly. Anthropic states the baseline: by default Claude Code asks for approval before running commands or modifying files, "and in practice users accept 93% of them anyway", which it calls approval fatigue. Use that 93% as a prompt for a check, not as a threshold. The audit takes one afternoon.
- Export the last 30 days of approval decisions for one action class: proposal text, approver, time shown, time decided, outcome.
- Compute three numbers: approval rate, median seconds from shown to decided, and the number of proposals where the approver edited or asked a question before deciding.
- Seed the next batch with flawed proposals you wrote and know are wrong (a finding closed on the wrong asset, a change broader than the ticket). Do this in staging or label them as drills. Count how many the approvers reject.
- Check capacity: proposals per week multiplied by the minutes a real review takes, divided by 60, against the hours your approvers have.
Example with invented figures, to show the arithmetic. In one week a cloud-posture agent produced 140 proposals. Approvers accepted 131 (93.6%), and the median time from shown to decided was 9 seconds. In a seeded batch of 6 flawed proposals they rejected 2, a catch rate of 33%. A real review of a configuration change takes about 4 minutes, so 140 proposals need 140 x 4 = 560 minutes, 9.3 hours of approver time. The approver pool has 3 hours a week set aside. The finding: the approval is a click, because the volume exceeds the time. The fix is not a better button. Cut the volume by moving the 90 low-risk proposals to "unattended, logged" under Q06, which leaves 50 proposals, 200 minutes, 3.3 hours: still 20 minutes over the 3 hours. Two ways to close the gap. Move 5 more of the lowest-risk proposals to "unattended, logged", which gives 45 x 4 = 180 minutes, exactly 3 hours with no slack. Or set aside 4 hours a week, which leaves 40 minutes for questions and escalations. Choose one, write it into the approval map, then repeat the seeded batch to see whether the catch rate rises.
Also try the enforcement side: from a test environment, call a high-risk tool without the approver step. The harness should refuse and log the refusal. The harness side of this is on approvals and checkpoints.
Sources
- OWASP, LLM06:2025 Excessive Agency, read 19 Nov 2026
- MITRE ATLAS data release 5.6.0, mitigation AML.M0029, read 19 Nov 2026
- OpenAI, A practical guide to building agents, guardrails section, read 19 Nov 2026
- EU AI Act, Article 14 Human oversight, read 19 Nov 2026
- Anthropic, How we built Claude Code auto mode (25 Mar 2026), read 19 Nov 2026
Last reviewed 19 Nov 2026