Choosing a model for a security agent

On 19 Nov 2026 the providers' pages list a window of about 1 million tokens for every current model, so a 40,000-token triage task fills 4% of it. Retirement dates, price changes, data retention terms and results on your own findings separate the models. A worked 120-trial test shows how.

Context windows and output limits as published

Every current model on the three providers' pages lists a window near 1 million tokens. A triage task of 40,000 input tokens (the assumption used on the cost page) uses about 4% of 1,048,576, and a 150,000-token fix task uses about 14%. Window size is therefore not what separates the current models for this work. The older Anthropic IDs still in use have 200K windows, where the same fix task uses 75%. Counts also differ by tokenizer: Anthropic's pricing page says Claude 4.7 and later models use a newer tokenizer that "produces approximately 30% more tokens for the same text", so a document that fits one provider's budget may not fit another's.

Providers' own models pages, read 19 Nov 2026, listed alphabetically. Prices are on the cost page. Anthropic prices claude-haiku-5-5 higher for prompts over 100,000 tokens; OpenAI's pricing page defines long context as more than 272K input tokens.
Provider and modelContext windowMaximum output
Anthropic, claude-fable-5-21M tokens128K tokens
Anthropic, claude-haiku-4-5-20251001200K tokens64K tokens
Anthropic, claude-haiku-5-51M tokens128K tokens
Anthropic, claude-opus-4-5-20251101200K tokens64K tokens
Anthropic, claude-opus-5-51M tokens128K tokens
Anthropic, claude-sonnet-4-5-20250929200K tokens64K tokens
Anthropic, claude-sonnet-4-61M tokens128K tokens
Anthropic, claude-sonnet-5-51M tokens128K tokens
Google, gemini-4-carbon1,048,576 (input and output together)Long decode up to 1M tokens
Google, gemini-4-hydrogen1,048,576 (input and output together)Long decode up to 1M tokens
Google, gemini-4-neon1,048,576 (input and output together)Long decode up to 1M tokens
OpenAI, gpt-6-luna1,050,000 (922,000 maximum input)128,000 tokens
OpenAI, gpt-6.1-sol1,050,000 (922,000 maximum input)128,000 tokens
OpenAI, gpt-6.1-terra1,050,000 (922,000 maximum input)128,000 tokens
OpenAI, gpt-6.2-astra1,050,000 (922,000 maximum input)128,000 tokens
OpenAI, gpt-6.2-sol1,050,000 (922,000 maximum input)128,000 tokens

Anthropic's models page lists a default effort setting per model: high for claude-fable-5-2 and claude-sonnet-5-5, medium for claude-opus-5-5 and claude-haiku-5-5. Google's pricing page bills thinking tokens as output. Both settings change how many tokens a run uses, so record the effort setting beside the model ID.

Dates already published that will change your model or its price

Each row is a date a provider has published for a model in the table above. "Not sooner than" is Anthropic's wording for an Active model: it is a floor, not a promise. OpenAI's deprecations table lists none of gpt-6-luna, gpt-6.1-sol, gpt-6.1-terra, gpt-6.2-astra or gpt-6.2-sol, and Google's pricing and model pages list no retirement date for the three Gemini 4 models.

Anthropic model deprecations page, OpenAI pricing page, Google Gemini API pricing page, all read 19 Nov 2026.
DateDays from 19 Nov 2026ModelWhat the provider's page states
15 Oct 2026Passedclaude-haiku-4-5-20251001Active, retirement not sooner than this date; still Active on 19 Nov 2026
24 Nov 20265claude-opus-4-5-20251101Active, retirement not sooner than this date
30 Nov 202611claude-sonnet-4-5-20250929Retires; Anthropic's recommended replacement is claude-sonnet-5-5
1 Dec 202612gemini-4-carbonInput $2.00 becomes $4.00 and output $10.00 becomes $20.00 per million tokens
17 Feb 202790claude-sonnet-4-6Active, retirement not sooner than this date
1 Mar 2027102gpt-6.2-astra and gpt-6.2-solUltrafast moves from 3 times to 6 times Standard (3 times applies through 28 Feb 2027)
22 Sep 2027307claude-opus-5-5Active, retirement not sooner than this date
28 Sep 2027313claude-sonnet-5-5Active, retirement not sooner than this date
7 Oct 2027322claude-haiku-5-5Active, retirement not sooner than this date
9 Nov 2027355claude-fable-5-2Active, retirement not sooner than this date

Anthropic says deprecated models "are likely to be less reliable than active models" and that its dates apply to Anthropic-operated platforms. Pin the dated model ID in the agent version, put a calendar entry 30 days before the earliest date above that touches a model you use, and rerun your test set on the replacement before the old model goes. The re-evaluation page lists what else triggers a rerun.

Data retention and residency differ by provider

Where a contract demands zero data retention or a fixed region, the provider's terms decide before any accuracy result does. The first column of retention figures below is the default for the standard paid API: Anthropic 30 days, Google 55 days and OpenAI up to 30 days. Confirm the plan you would buy, since several terms depend on the plan or on approval.

Anthropic privacy center, Cyber Verification Program announcement and data residency page; Google Gemini API usage policies and Additional Terms; OpenAI data controls and pricing pages. All read 19 Nov 2026.
ProviderDefault retention, standard paid APIOther retention terms statedResidency terms stated
AnthropicInputs and outputs are deleted from Anthropic's backend within 30 days of receipt or generation ("we automatically delete inputs and outputs on our backend within 30 days of receipt or generation")Organizations enrolled in the Cyber Verification Program must allow data retention: "data retention is required for organizations enrolled in the program". Anthropic's data residency page lists zero data retention as an eligible data-handling optioninference_geo set to "us" keeps inference in US infrastructure at a 1.1 times price; the default "global" may run anywhere
GooglePaid Services: prompts and responses are logged for 55 days ("fifty-five (55) days") to detect violations of the Prohibited Use Policy; the Additional Terms say only "a limited period of time"Paid Services: prompts and responses are not used to improve Google products. Unpaid Services: Google uses submitted content to improve its products. Grounding with Google Search stores prompts and outputs for 30 daysPaid-service data "may be stored transiently or cached in any country in which Google or its agents maintain facilities"
OpenAIAbuse monitoring logs of prompts and responses are kept up to 30 daysZero Data Retention and Modified Abuse Monitoring need OpenAI's prior approval; Zero Data Retention treats the store parameter as falseRegional processing endpoints carry a 10% price uplift for models released on or after 5 Mar 2026

Cyber safeguards decide whether the model finishes the work

Anthropic states that its generally available models "have conservative cyber safeguards that block most cyber work" and "can continue to be used for tasks such as code review, patching known issues, vulnerability finding in owned source code, and triage of security alerts" (Cyber Verification Program announcement, vendor statement). Enrollment brings the retention term in the table above.

A block rate is a property of a model, a date and a task, and it moves from day to day. A security-agent vendor's day-by-day series, with the effect of retries, is on measuring an agent after it ships. Three consequences carry over to model choice: count a blocked run as its own outcome, not as a wrong answer; decide in advance how many retries are allowed; and test the fallback model you would switch to, because a run that falls back to a different model changes your results without a warning.

A worked comparison of two models, 120 trials each

The results below are invented to show the arithmetic. Only the prices are published: Model H is priced like claude-haiku-5-5 ($0.10 and $0.50 per million tokens) and Model S like claude-sonnet-5-5 ($2 and $10). Both ran the same harness, the same prompt and the same 40 triage findings, 3 trials each, 120 trials. A triage trial uses 40,000 input and 4,000 output tokens, so $0.006 on H and $0.12 on S. Assumptions for people: every trial takes 3 minutes of review, and each trial that is not accepted costs 15 minutes more, at $100 an hour.

Invented results, published prices read 19 Nov 2026. Review time is 120 x 3 minutes plus 15 minutes for each trial not accepted (42 for H, 18 for S).
MeasureModel HModel S
Accepted trials (outcome verified in the system)78 of 120 (65%)102 of 120 (85%)
Blocked runs, counted among the not accepted66
Chance one finding passes all 3 trials if the rate holds (rate cubed, as in AI agent evals)27%61%
Token cost for 120 trials$0.72$14.40
Token cost per accepted trial$0.009$0.141
Review and rework time990 minutes630 minutes
Total cost at $100 an hour$1,650.72$1,064.40
Cost per accepted trial$21.16$10.44

The model priced 20 times higher per token costs about half as much per accepted result. The break-even is small: S has 24 fewer failures and costs $13.68 more in tokens, so S is cheaper as soon as one failure costs more than $0.57, about 20 seconds of review time. Measure the real rates from your own trials rather than computing them.

Running the test

  1. Fix the set: 30 to 50 findings with a senior analyst's answer. Anthropic's eval guide says "20-50 simple tasks drawn from real failures is a great start". The method is on evaluating on your findings.
  2. Hold the harness, prompt and tools constant. Change only the model ID and its effort setting. Anthropic: "When we evaluate 'an agent,' we're evaluating the harness and the model working together."
  3. Run 3 trials per finding, each from a clean start.
  4. Record per model: accepted, blocked, token cost per accepted trial, review minutes.
  5. Rerun on every date in the table above that touches your model, and on every model, prompt or tool change.

Public benchmarks do not replace this. Anthropic's guide reports that SWE-Bench Verified began the year at 30% and that frontier models are now "nearing saturation at >80%", which leaves little room to separate them. The benchmarks section says what each public benchmark measures, and AI agent evals covers the grader types. Keep the model ID, effort setting and date in the run log (what to keep in a run log).

Sources

  1. Anthropic, Claude pricing (tokenizer note, Haiku 5.5 tiers), read 19 Nov 2026
  2. Anthropic, Models overview (context, output, effort, retirement), read 19 Nov 2026
  3. Anthropic, Model deprecations, read 19 Nov 2026
  4. OpenAI, API pricing, read 19 Nov 2026
  5. OpenAI, gpt-6.2-sol model page (context window and output limits; same figures on the gpt-6-luna, gpt-6.1-sol, gpt-6.1-terra and gpt-6.2-astra pages), read 19 Nov 2026
  6. OpenAI, Deprecations, read 19 Nov 2026
  7. Google, Gemini API pricing, read 19 Nov 2026
  8. Google, Gemini 4 Carbon model page (token limits), read 19 Nov 2026
  9. Anthropic, Expanding the Cyber Verification Program, read 19 Nov 2026
  10. Anthropic, Demystifying evals for AI agents, read 19 Nov 2026
  11. Anthropic, Data residency, read 19 Nov 2026
  12. OpenAI, Data controls in the OpenAI platform, read 19 Nov 2026
  13. Google, Gemini API Additional Terms of Service, read 19 Nov 2026
  14. Anthropic privacy center, How long do you store my organization's data, read 19 Nov 2026
  15. Google, Gemini API usage policies (How we handle data: 55-day retention), read 19 Nov 2026
  16. OpenAI, gpt-6.1-sol model page (context window and output limits), read 19 Nov 2026

Last reviewed 19 Nov 2026