Questions to ask AI agent vendors

Before buying a security agent, get written answers on who controls tools, identity, approvals, logs, update timing and known limits. CISA's joint guidance of 15 April 2024 says to "Consider deployment environment security requirements when developing contracts for AI system products or services." NCSC asks suppliers for preview access and versioned APIs.

The two questions

  • Q19. If the agent is bought or hosted by a supplier, what do you control (tools, permissions, approval gates, logs, update timing) and what does the supplier control?
  • Q20. If you build, which parts (model, harness, tools, sandbox) can be swapped later, and what has the supplier told you about known limitations and failure modes?

Where the requirements come from

Requirements read 19 Nov 2026. The question numbers refer to the supplier form below, which is this checklist's wording.
Quoted requirementSourceWhat it supports
"Consider deployment environment security requirements when developing contracts for AI system products or services."CISA and partners, Deploying AI Systems Securely, v1.0, April 2024Every row of the form below has contract wording, so the answer is in the contract and not only in the sales call
"Require the primary developer of the AI system to provide a threat model for their system."Same CISA documentQuestion 8
"Identify the roles and responsibilities for each stakeholder along with how they are accountable for fulfilling them"Same CISA documentThe responsibility split in Q19
"Prepare for automated rollbacks and use advanced deployments with a human-in-the-loop as a failsafe"Same CISA documentQuestion 6
"Collect logs to cover inputs, outputs, intermediate states, and errors"Same CISA documentQuestion 4
"You provide high-quality audit logs and other security features or information to customers and users at no extra charge, to enable their incident response processes."NCSC, secure deploymentQuestion 4: logs are not an upsell
"You support users to evaluate and respond to model changes (for example by providing preview access and versioned APIs)."NCSC, secure operation and maintenanceQuestion 5
"you are clear to your users about known limitations or potential failure modes"NCSC, secure deploymentQuestion 7
"Risks and benefits are mapped for all components of the AI system including third-party software and data."NIST AI RMF 1.0, MAP 4Your own map of what the supplier's parts do

The supplier form and its contract wording

Nine questions. Rows 1 to 6 answer the Q19 split of control (tools, identity, approvals, logs, update timing, stopping). Row 7 answers the known-limitations half of Q20. Rows 8 and 9 come from the CISA threat-model clause and the NIST MAP 4 mapping of third-party software and data. The accept and escalate columns are this checklist's tests.
No.Ask the supplierAccept if the answer saysEscalate ifContract wording (brackets are values to set)
1Can we remove or narrow tools, and are tool lists shown per tenant?Per-tool on and off switches, each tool with one system and one kind of change (scope of change)A general shell or URL-fetch tool cannot be removedSupplier delivers a list of tools, each with system, kind of change (read, comment, ticket, configure, block, delete) and limits. Customer may disable any tool.
2Which identity does the agent use in our systems, and who creates it?A role we create and can revoke; no shared supplier accountThe supplier holds a standing admin credential for our tenantAgent acts only through roles the Customer creates. Customer may revoke a role at any time; revocation takes effect within [5] minutes.
3Which actions can we set to need approval, by what rule, and what is the default?A setting we control by action type, with a documented default (human approval points)Approval is on only for the supplier's choice of actionsCustomer sets which actions need human approval. Supplier changes no default without [30] days' notice.
4Do we get step-level logs we can export, in what fields, kept how long, at what price?Field list covering inputs, outputs, intermediate states and errors; export by API; retention stated; included in the price (evidence and audit trail)Logs are summaries only, or cost extraSupplier provides step-level logs (inputs, outputs, intermediate states, errors) by export API, retained [180] days, at no extra charge.
5Can we pin a version or preview an update, and how much notice do we get?A pinned version and a written notice period of at least the time our re-test takes (re-evaluating after changes)Model changes apply with no noticeSupplier gives [10] business days' notice of any change to a model, prompt or tool, with preview access in a test tenant. Customer may pin the previous version for [90] days.
6Can we stop the agent and undo its changes without the supplier?A customer-side stop control and a list of what each tool can undo (kill switch and rollback)Stopping needs a support ticketCustomer can halt all agent actions without contacting Supplier. Supplier lists which actions each tool can reverse.
7What failure modes and limitations do you document, and which did you find in your own tests?A written limitations page with at least one failure mode per toolNo document exists, or it lists noneSupplier maintains a written list of known limitations and failure modes and sends updates within [14] days of finding one.
8Can you provide a threat model for the system?A document we can read under NDAThe supplier declinesSupplier provides a threat model for the system on request.
9Where does our data go, and is it used to train or tune a model?Named processing locations and a written no-training termSilence on training useSupplier names the locations where Customer data is processed and does not use it to train or tune a model without Customer's written consent.

Worked example: how much notice is enough

The notice period in row 5 comes from your own re-test, not from a round number. Take the 40-case, 3-trial set from success criteria and test sets: 120 runs. At 3 minutes a run in the sandbox that is 360 minutes, 6 hours. Reading 20 transcripts at 5 minutes each adds 100 minutes. A decision meeting adds 60 minutes. Total: 520 minutes, about 8 hours 40 minutes, so one working day if a person starts at the beginning of it.

A supplier that gives preview access and 10 business days' notice leaves 9 spare days. A supplier that applies model changes with no notice leaves none: the test runs after the change reached production, which is the case NCSC's preview-access wording is written to prevent. Logs work the same way. If your incident reviews reach back 180 days and the supplier keeps logs for 30, you need a daily export to your own store, and the contract should say so.

If you build (Q20)

Q20 is about which parts can be replaced. Anthropic describes one split: a session ("the append-only log of everything that happened"), a harness ("the loop that calls Claude and routes Claude's tool calls to the relevant infrastructure") and a sandbox, which "allows the implementation of each to be swapped without disturbing the others" (Scaling Managed Agents, 8 Apr 2026). Anthropic also advises: "Frameworks can help you get started quickly, but don't hesitate to reduce abstraction layers and build with basic components as you move to production" (Building effective agents).

Interfaces to record for Q20. Margins and counts are yours to set. The cost side of building is in the build vs buy guide; the parts are named in the agent harness section.
PartWrite downSwap test
ModelModel name, version, and the prompt that depends on itChange the model in staging and run the regression set; accept if the pass rate is within the margin you wrote before the run and no new out-of-scope call appears
HarnessThe loop, retry and stop limits, and the format of tool callsRun the same 120 trials on the old and the new loop
ToolsThe register from scope of changeReplace one tool's backend; the tool name, inputs and outputs stay
SandboxThe copy's addresses and how it is rebuiltRebuild it from a clean base and rerun the set

How to test it on your own systems

  1. Ask the supplier to show each of rows 1 to 6 of the form in a trial tenant and record who set the value.
  2. Ask for the last three update notices and check whether any change was applied before you could test it.
  3. Compare the supplier's limitations document with what your own test set found; a failure you found that the document omits is a question for the supplier.

Sources

  1. NSA, CISA, FBI and partners, Deploying AI Systems Securely, Cybersecurity Information Sheet v1.0, April 2024, read 19 Nov 2026
  2. CISA, Joint Guidance on Deploying AI Systems Securely (alert of 15 Apr 2024), read 19 Nov 2026
  3. NIST AI RMF 1.0 (NIST AI 100-1), MAP 4, read 19 Nov 2026
  4. NCSC, Guidelines for secure AI system development, secure deployment, read 19 Nov 2026
  5. NCSC, Guidelines for secure AI system development, secure operation and maintenance, read 19 Nov 2026
  6. Anthropic, Scaling Managed Agents (8 Apr 2026), read 19 Nov 2026
  7. Anthropic, Building effective agents, read 19 Nov 2026

Last reviewed 19 Nov 2026