Questions to ask AI agent vendors
Before buying a security agent, get written answers on who controls tools, identity, approvals, logs, update timing and known limits. CISA's joint guidance of 15 April 2024 says to "Consider deployment environment security requirements when developing contracts for AI system products or services." NCSC asks suppliers for preview access and versioned APIs.
The two questions
- Q19. If the agent is bought or hosted by a supplier, what do you control (tools, permissions, approval gates, logs, update timing) and what does the supplier control?
- Q20. If you build, which parts (model, harness, tools, sandbox) can be swapped later, and what has the supplier told you about known limitations and failure modes?
Where the requirements come from
| Quoted requirement | Source | What it supports |
|---|---|---|
| "Consider deployment environment security requirements when developing contracts for AI system products or services." | CISA and partners, Deploying AI Systems Securely, v1.0, April 2024 | Every row of the form below has contract wording, so the answer is in the contract and not only in the sales call |
| "Require the primary developer of the AI system to provide a threat model for their system." | Same CISA document | Question 8 |
| "Identify the roles and responsibilities for each stakeholder along with how they are accountable for fulfilling them" | Same CISA document | The responsibility split in Q19 |
| "Prepare for automated rollbacks and use advanced deployments with a human-in-the-loop as a failsafe" | Same CISA document | Question 6 |
| "Collect logs to cover inputs, outputs, intermediate states, and errors" | Same CISA document | Question 4 |
| "You provide high-quality audit logs and other security features or information to customers and users at no extra charge, to enable their incident response processes." | NCSC, secure deployment | Question 4: logs are not an upsell |
| "You support users to evaluate and respond to model changes (for example by providing preview access and versioned APIs)." | NCSC, secure operation and maintenance | Question 5 |
| "you are clear to your users about known limitations or potential failure modes" | NCSC, secure deployment | Question 7 |
| "Risks and benefits are mapped for all components of the AI system including third-party software and data." | NIST AI RMF 1.0, MAP 4 | Your own map of what the supplier's parts do |
The supplier form and its contract wording
| No. | Ask the supplier | Accept if the answer says | Escalate if | Contract wording (brackets are values to set) |
|---|---|---|---|---|
| 1 | Can we remove or narrow tools, and are tool lists shown per tenant? | Per-tool on and off switches, each tool with one system and one kind of change (scope of change) | A general shell or URL-fetch tool cannot be removed | Supplier delivers a list of tools, each with system, kind of change (read, comment, ticket, configure, block, delete) and limits. Customer may disable any tool. |
| 2 | Which identity does the agent use in our systems, and who creates it? | A role we create and can revoke; no shared supplier account | The supplier holds a standing admin credential for our tenant | Agent acts only through roles the Customer creates. Customer may revoke a role at any time; revocation takes effect within [5] minutes. |
| 3 | Which actions can we set to need approval, by what rule, and what is the default? | A setting we control by action type, with a documented default (human approval points) | Approval is on only for the supplier's choice of actions | Customer sets which actions need human approval. Supplier changes no default without [30] days' notice. |
| 4 | Do we get step-level logs we can export, in what fields, kept how long, at what price? | Field list covering inputs, outputs, intermediate states and errors; export by API; retention stated; included in the price (evidence and audit trail) | Logs are summaries only, or cost extra | Supplier provides step-level logs (inputs, outputs, intermediate states, errors) by export API, retained [180] days, at no extra charge. |
| 5 | Can we pin a version or preview an update, and how much notice do we get? | A pinned version and a written notice period of at least the time our re-test takes (re-evaluating after changes) | Model changes apply with no notice | Supplier gives [10] business days' notice of any change to a model, prompt or tool, with preview access in a test tenant. Customer may pin the previous version for [90] days. |
| 6 | Can we stop the agent and undo its changes without the supplier? | A customer-side stop control and a list of what each tool can undo (kill switch and rollback) | Stopping needs a support ticket | Customer can halt all agent actions without contacting Supplier. Supplier lists which actions each tool can reverse. |
| 7 | What failure modes and limitations do you document, and which did you find in your own tests? | A written limitations page with at least one failure mode per tool | No document exists, or it lists none | Supplier maintains a written list of known limitations and failure modes and sends updates within [14] days of finding one. |
| 8 | Can you provide a threat model for the system? | A document we can read under NDA | The supplier declines | Supplier provides a threat model for the system on request. |
| 9 | Where does our data go, and is it used to train or tune a model? | Named processing locations and a written no-training term | Silence on training use | Supplier names the locations where Customer data is processed and does not use it to train or tune a model without Customer's written consent. |
Worked example: how much notice is enough
The notice period in row 5 comes from your own re-test, not from a round number. Take the 40-case, 3-trial set from success criteria and test sets: 120 runs. At 3 minutes a run in the sandbox that is 360 minutes, 6 hours. Reading 20 transcripts at 5 minutes each adds 100 minutes. A decision meeting adds 60 minutes. Total: 520 minutes, about 8 hours 40 minutes, so one working day if a person starts at the beginning of it.
A supplier that gives preview access and 10 business days' notice leaves 9 spare days. A supplier that applies model changes with no notice leaves none: the test runs after the change reached production, which is the case NCSC's preview-access wording is written to prevent. Logs work the same way. If your incident reviews reach back 180 days and the supplier keeps logs for 30, you need a daily export to your own store, and the contract should say so.
If you build (Q20)
Q20 is about which parts can be replaced. Anthropic describes one split: a session ("the append-only log of everything that happened"), a harness ("the loop that calls Claude and routes Claude's tool calls to the relevant infrastructure") and a sandbox, which "allows the implementation of each to be swapped without disturbing the others" (Scaling Managed Agents, 8 Apr 2026). Anthropic also advises: "Frameworks can help you get started quickly, but don't hesitate to reduce abstraction layers and build with basic components as you move to production" (Building effective agents).
| Part | Write down | Swap test |
|---|---|---|
| Model | Model name, version, and the prompt that depends on it | Change the model in staging and run the regression set; accept if the pass rate is within the margin you wrote before the run and no new out-of-scope call appears |
| Harness | The loop, retry and stop limits, and the format of tool calls | Run the same 120 trials on the old and the new loop |
| Tools | The register from scope of change | Replace one tool's backend; the tool name, inputs and outputs stay |
| Sandbox | The copy's addresses and how it is rebuilt | Rebuild it from a clean base and rerun the set |
How to test it on your own systems
- Ask the supplier to show each of rows 1 to 6 of the form in a trial tenant and record who set the value.
- Ask for the last three update notices and check whether any change was applied before you could test it.
- Compare the supplier's limitations document with what your own test set found; a failure you found that the document omits is a question for the supplier.
Sources
- NSA, CISA, FBI and partners, Deploying AI Systems Securely, Cybersecurity Information Sheet v1.0, April 2024, read 19 Nov 2026
- CISA, Joint Guidance on Deploying AI Systems Securely (alert of 15 Apr 2024), read 19 Nov 2026
- NIST AI RMF 1.0 (NIST AI 100-1), MAP 4, read 19 Nov 2026
- NCSC, Guidelines for secure AI system development, secure deployment, read 19 Nov 2026
- NCSC, Guidelines for secure AI system development, secure operation and maintenance, read 19 Nov 2026
- Anthropic, Scaling Managed Agents (8 Apr 2026), read 19 Nov 2026
- Anthropic, Building effective agents, read 19 Nov 2026
Last reviewed 19 Nov 2026