CTIBench

CTIBench (arXiv 2406.07599, NeurIPS 2024) tests language models on 2,500 threat intelligence questions and four practical tasks. In the paper, gpt-4-turbo reached 71.0% on the questions and 72.0% on CVE-to-CWE mapping, and named the right actor for 52% of 50 reports. Table read 19 Nov 2026.

What the five datasets contain

Four authors (Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, Nidhi Rastogi) posted arXiv 2406.07599 on 11 Jun 2024; the current version is v3 of 11 Nov 2024. The project page lists it as a NeurIPS 2024 Datasets and Benchmarks spotlight. The repository README counts 4,610 released examples across the five tasks (2,500 + 1,000 + 1,000 + 60 + 50), plus a separate 1,000-item 2021 comparison split for CWE mapping, which makes 5,610 rows in the data folder.

Sizes from the paper's Section 3 and the repository README, read 19 Nov 2026. The paper and the README both state 2,500 for CTI-MCQ; the paper does not explain the 100-question gap in its source breakdown.
DatasetTaskSize and source
CTI-MCQMultiple-choice questions on standards, threat identification, detection and mitigation2,500 questions, written with GPT-4o from MITRE, CWE, NIST, GDPR and STIX/TAXII material and then checked by hand; the paper lists 1,578 from MITRE, 750 from CWE, 40 from public quizzes and 32 from standards, which add to 2,400
CTI-RCMMap a CVE description to its CWE1,000 CVEs from 2024 sampled from the NVD, with their NVD mappings as the answer key
CTI-VSPPredict the CVSS v3 vector string from a CVE description1,000 descriptions from 2024 with their CVSS v3 strings
CTI-ATEExtract MITRE ATT&CK technique IDs from a description of malware behavior60 malware entries from MITRE ATT&CK (30 reported in 2024, 30 earlier), covering 397 unique techniques; sub-techniques excluded
CTI-TAAName the threat actor behind a vendor report50 vendor APT reports with actor and campaign names replaced by placeholders

The CTI-RCM task is shaped like this paper example: the description of CVE-2018-15506 says BubbleUPnP 0.9 update 30 is "vulnerable to an XML External Entity Processing (XXE) attack", and the answer key is CWE-611. A model is right only if its last line names that CWE ID.

How each task is graded

  • CTI-MCQ and CTI-RCM: accuracy, one answer per item.
  • CTI-VSP: the model's vector string is converted to a numeric CVSS v3.1 rating with the Python cvss library, and the result is the mean absolute deviation (MAD) from the NVD rating on the 0 to 10 scale. Lower is better.
  • CTI-ATE: F1 over technique IDs. The text names Micro-F1; the results table header says Macro-F1.
  • CTI-TAA: a human checks each answer against the actor's aliases and files it as correct, plausible (a related actor in a similar group) or incorrect. The table reports correct, and correct plus plausible.

Every model ran zero-shot at temperature 0 and top_p 1 with a prompt telling it to act as a cybersecurity expert. The paper names the models as ChatGPT-3.5 (gpt-3.5-turbo), ChatGPT-4 (gpt-4-turbo), Gemini-1.5, LLAMA3-70B and LLAMA3-8B. The repository's model-prediction notebook maps them to gemini-1.5-pro (Vertex AI), meta/meta-llama-3-70b-instruct and meta/meta-llama-3-8b-instruct (Replicate), so the Gemini row is Gemini 1.5 Pro and the two Llama rows are the Instruct variants; no dated build is given for any of them.

Published results

Table 1 of arXiv 2406.07599 v3 (June to November 2024 models). The project page at maveryn.github.io/cti-bench shows the same five rows, read 19 Nov 2026. Model identifiers in brackets are from the repository notebook.
Model (as named in the paper)CTI-MCQ accuracyCTI-RCM accuracyCTI-VSP MAD (lower is better)CTI-ATE F1CTI-TAA correctCTI-TAA correct or plausible
ChatGPT-3.5 (gpt-3.5-turbo)54.1%67.2%1.570.310844%62%
ChatGPT-4 (gpt-4-turbo)71.0%72.0%1.310.638852%86%
Gemini-1.5 (gemini-1.5-pro)65.4%66.6%1.090.461238%74%
LLAMA3-70B (meta-llama-3-70b-instruct)65.7%65.9%1.830.472052%80%
LLAMA3-8B (meta-llama-3-8b-instruct)61.3%44.7%1.910.156228%36%

The paper's other findings, each from its own table. On CTI-MCQ, 293 questions were missed by all five models, mostly on mitigation plans and adversary tools. Asking the models to reason before answering raised gpt-4-turbo from 71.00% to 71.84% while its output grew from about 105 thousand to 618 thousand tokens, and lowered LLAMA3-8B from 61.32% to 55.80%. On CTI-TAA, four of five models did better on reports from before their knowledge cutoff: gpt-4-turbo's correct rate fell from 58.06% to 42.10% on newer reports, which the authors read as memorization helping.

Later papers have used CTIBench's four core tasks on newer models. Tian et al. (arXiv 2610.20284, 14 Oct 2026) report untuned baselines for two 2024 open models; the authors' successor benchmark AthenaBench (arXiv 2511.01144, v2 14 Feb 2026) uses new data and different metrics, so its figures do not compare with the table above.

Table 1 of arXiv 2610.20284 (14 Oct 2026), which evaluates four of the five CTIBench tasks and omits CTI-TAA. Read 19 Nov 2026.
Model (as named by Tian et al.)CTI-MCQ accuracyCTI-RCM accuracyCTI-VSP MAD (lower is better)CTI-ATE F1
Qwen2.5-7B-Instruct, untuned63.28%55.79%1.69700.1995
Llama-3.1-8B-Instruct, untuned53.96%57.00%1.86170.1763
Qwen2.5-7B-Instruct, tuned on 2,500 selected examples (their method)66.24%59.33%1.52380.2417
Llama-3.1-8B-Instruct, tuned on 2,500 selected examples (their method)59.23%61.44%1.64720.2284

AthenaBench evaluates GPT-5, Gemini-2.5 (Flash and Pro), GPT-4o and others on 3,000 knowledge questions and five further tasks. GPT-5 reaches 92.0% on its knowledge test and 71.6% on root-cause mapping; its accuracy for severity prediction is one minus the mean deviation divided by 7.7, not the MAD used by CTIBench.

Which gaps in the table are real

Accuracy on n items has a 95% interval of about 1.96 times the square root of p(1-p)/n (normal approximation, worked here, not in the paper).

Intervals computed on 19 Nov 2026 from the paper's counts.
ResultItemsIntervalWhat it means
CTI-MCQ, 71.0%2,500plus or minus 1.8 points1,775 right; the 5.6-point gap to Gemini-1.5 at 65.4% is outside the interval
CTI-RCM, 72.0%1,000plus or minus 2.8 points720 right and 280 wrong; Gemini-1.5 (66.6%) and LLAMA3-70B (65.9%) differ by 0.7 points, inside the noise
CTI-TAA, 52%50plus or minus 13.8 points26 right; Gemini-1.5 at 38% is 19 right, plus or minus 13.5, so the two intervals overlap
CTI-ATE, F1 0.638860 entriesnot given in the paperOnly 60 entries and no interval in the paper; the 0.0108 gap between Gemini-1.5 (0.4612) and LLAMA3-70B (0.4720) should not be read as a difference

What it does not measure

  • An agent acting on systems. Every task is one prompt and one answer on fixed text; there are no tools, no approvals and no side effects.
  • Skill on your findings. CTI-VSP asks for a CVSS vector from a description; a MAD of 1.09 to 1.91 on a 0 to 10 scale says nothing about what a finding is worth in your environment.
  • Frontier models after mid-2024. The paper's five models date from 2024 and the dataset has been public since June 2024, so a model trained later may have seen it; the only later CTIBench figures are the two small open models above.
  • Non-English or broader intelligence work. The authors state the study covers "a limited subset of tasks", is restricted to English, and that questions drawn from public quizzes "may introduce some bias".

How a security team would use it

Use it as a cheap screen on the model behind an agent, before spending time on an agent test. Run your candidate on the released files with the paper's prompt, then compare against the intervals above, not the point values.

Duties are this site's reading of each task; the figures are the paper's.
TaskAgent duty it speaks toReading the paper's figures
CTI-MCQVocabulary of the intelligence the agent readsHighest reported 71.0%: about 29 in 100 questions wrong
CTI-RCMTagging a finding with its weakness classHighest reported 72.0%: 280 of 1,000 mapped to the wrong CWE
CTI-VSPA first-pass severity guess to be checkedClosest reported MAD 1.09: a typical miss of over a point on a 0 to 10 scale
CTI-ATEMapping alert text to ATT&CK techniquesF1 between 0.1562 and 0.6388
CTI-TAASuggesting an actor for an analyst to confirmCorrect on 26 of 50 at most; treat output as a lead for review, never as a finding

If a model falls below ChatGPT-3.5's figures in the paper on the task your agent performs, stop there. If it passes, the next test is your own: build a set from your closed tickets as in evaluating a security agent on your findings, and keep the model choice tied to the success criteria and test sets you set before testing. The same family of questions, with images and SOC data, appears in CyberSecEval 4's CyberSOCEval; the benchmarks index lists the rest, and choosing a model for a security agent sets this screen among the other factors.

Sources

  1. Alam et al., CTIBench, arXiv 2406.07599 (v3, 11 Nov 2024), read 19 Nov 2026
  2. CTIBench full text: Sections 3 to 5, Appendix E, Table 1 to 4, read 19 Nov 2026
  3. CTIBench repository README: dataset sizes and evaluated models, read 19 Nov 2026
  4. CTIBench project page: results table, NeurIPS 2024 spotlight, read 19 Nov 2026
  5. Tian et al., CTI-TDS instruction tuning evaluated on CTIBench, arXiv 2610.20284 (14 Oct 2026), read 19 Nov 2026
  6. Alam et al., AthenaBench, arXiv 2511.01144 (v2, 14 Feb 2026), read 19 Nov 2026

Last reviewed 19 Nov 2026