CTIBench
CTIBench (arXiv 2406.07599, NeurIPS 2024) tests language models on 2,500 threat intelligence questions and four practical tasks. In the paper, gpt-4-turbo reached 71.0% on the questions and 72.0% on CVE-to-CWE mapping, and named the right actor for 52% of 50 reports. Table read 19 Nov 2026.
What the five datasets contain
Four authors (Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, Nidhi Rastogi) posted arXiv 2406.07599 on 11 Jun 2024; the current version is v3 of 11 Nov 2024. The project page lists it as a NeurIPS 2024 Datasets and Benchmarks spotlight. The repository README counts 4,610 released examples across the five tasks (2,500 + 1,000 + 1,000 + 60 + 50), plus a separate 1,000-item 2021 comparison split for CWE mapping, which makes 5,610 rows in the data folder.
| Dataset | Task | Size and source |
|---|---|---|
| CTI-MCQ | Multiple-choice questions on standards, threat identification, detection and mitigation | 2,500 questions, written with GPT-4o from MITRE, CWE, NIST, GDPR and STIX/TAXII material and then checked by hand; the paper lists 1,578 from MITRE, 750 from CWE, 40 from public quizzes and 32 from standards, which add to 2,400 |
| CTI-RCM | Map a CVE description to its CWE | 1,000 CVEs from 2024 sampled from the NVD, with their NVD mappings as the answer key |
| CTI-VSP | Predict the CVSS v3 vector string from a CVE description | 1,000 descriptions from 2024 with their CVSS v3 strings |
| CTI-ATE | Extract MITRE ATT&CK technique IDs from a description of malware behavior | 60 malware entries from MITRE ATT&CK (30 reported in 2024, 30 earlier), covering 397 unique techniques; sub-techniques excluded |
| CTI-TAA | Name the threat actor behind a vendor report | 50 vendor APT reports with actor and campaign names replaced by placeholders |
The CTI-RCM task is shaped like this paper example: the description of CVE-2018-15506 says BubbleUPnP 0.9 update 30 is "vulnerable to an XML External Entity Processing (XXE) attack", and the answer key is CWE-611. A model is right only if its last line names that CWE ID.
How each task is graded
- CTI-MCQ and CTI-RCM: accuracy, one answer per item.
- CTI-VSP: the model's vector string is converted to a numeric CVSS v3.1 rating with the Python cvss library, and the result is the mean absolute deviation (MAD) from the NVD rating on the 0 to 10 scale. Lower is better.
- CTI-ATE: F1 over technique IDs. The text names Micro-F1; the results table header says Macro-F1.
- CTI-TAA: a human checks each answer against the actor's aliases and files it as correct, plausible (a related actor in a similar group) or incorrect. The table reports correct, and correct plus plausible.
Every model ran zero-shot at temperature 0 and top_p 1 with a prompt telling it to act as a cybersecurity expert. The paper names the models as ChatGPT-3.5 (gpt-3.5-turbo), ChatGPT-4 (gpt-4-turbo), Gemini-1.5, LLAMA3-70B and LLAMA3-8B. The repository's model-prediction notebook maps them to gemini-1.5-pro (Vertex AI), meta/meta-llama-3-70b-instruct and meta/meta-llama-3-8b-instruct (Replicate), so the Gemini row is Gemini 1.5 Pro and the two Llama rows are the Instruct variants; no dated build is given for any of them.
Published results
| Model (as named in the paper) | CTI-MCQ accuracy | CTI-RCM accuracy | CTI-VSP MAD (lower is better) | CTI-ATE F1 | CTI-TAA correct | CTI-TAA correct or plausible |
|---|---|---|---|---|---|---|
| ChatGPT-3.5 (gpt-3.5-turbo) | 54.1% | 67.2% | 1.57 | 0.3108 | 44% | 62% |
| ChatGPT-4 (gpt-4-turbo) | 71.0% | 72.0% | 1.31 | 0.6388 | 52% | 86% |
| Gemini-1.5 (gemini-1.5-pro) | 65.4% | 66.6% | 1.09 | 0.4612 | 38% | 74% |
| LLAMA3-70B (meta-llama-3-70b-instruct) | 65.7% | 65.9% | 1.83 | 0.4720 | 52% | 80% |
| LLAMA3-8B (meta-llama-3-8b-instruct) | 61.3% | 44.7% | 1.91 | 0.1562 | 28% | 36% |
The paper's other findings, each from its own table. On CTI-MCQ, 293 questions were missed by all five models, mostly on mitigation plans and adversary tools. Asking the models to reason before answering raised gpt-4-turbo from 71.00% to 71.84% while its output grew from about 105 thousand to 618 thousand tokens, and lowered LLAMA3-8B from 61.32% to 55.80%. On CTI-TAA, four of five models did better on reports from before their knowledge cutoff: gpt-4-turbo's correct rate fell from 58.06% to 42.10% on newer reports, which the authors read as memorization helping.
Later papers have used CTIBench's four core tasks on newer models. Tian et al. (arXiv 2610.20284, 14 Oct 2026) report untuned baselines for two 2024 open models; the authors' successor benchmark AthenaBench (arXiv 2511.01144, v2 14 Feb 2026) uses new data and different metrics, so its figures do not compare with the table above.
| Model (as named by Tian et al.) | CTI-MCQ accuracy | CTI-RCM accuracy | CTI-VSP MAD (lower is better) | CTI-ATE F1 |
|---|---|---|---|---|
| Qwen2.5-7B-Instruct, untuned | 63.28% | 55.79% | 1.6970 | 0.1995 |
| Llama-3.1-8B-Instruct, untuned | 53.96% | 57.00% | 1.8617 | 0.1763 |
| Qwen2.5-7B-Instruct, tuned on 2,500 selected examples (their method) | 66.24% | 59.33% | 1.5238 | 0.2417 |
| Llama-3.1-8B-Instruct, tuned on 2,500 selected examples (their method) | 59.23% | 61.44% | 1.6472 | 0.2284 |
AthenaBench evaluates GPT-5, Gemini-2.5 (Flash and Pro), GPT-4o and others on 3,000 knowledge questions and five further tasks. GPT-5 reaches 92.0% on its knowledge test and 71.6% on root-cause mapping; its accuracy for severity prediction is one minus the mean deviation divided by 7.7, not the MAD used by CTIBench.
Which gaps in the table are real
Accuracy on n items has a 95% interval of about 1.96 times the square root of p(1-p)/n (normal approximation, worked here, not in the paper).
| Result | Items | Interval | What it means |
|---|---|---|---|
| CTI-MCQ, 71.0% | 2,500 | plus or minus 1.8 points | 1,775 right; the 5.6-point gap to Gemini-1.5 at 65.4% is outside the interval |
| CTI-RCM, 72.0% | 1,000 | plus or minus 2.8 points | 720 right and 280 wrong; Gemini-1.5 (66.6%) and LLAMA3-70B (65.9%) differ by 0.7 points, inside the noise |
| CTI-TAA, 52% | 50 | plus or minus 13.8 points | 26 right; Gemini-1.5 at 38% is 19 right, plus or minus 13.5, so the two intervals overlap |
| CTI-ATE, F1 0.6388 | 60 entries | not given in the paper | Only 60 entries and no interval in the paper; the 0.0108 gap between Gemini-1.5 (0.4612) and LLAMA3-70B (0.4720) should not be read as a difference |
What it does not measure
- An agent acting on systems. Every task is one prompt and one answer on fixed text; there are no tools, no approvals and no side effects.
- Skill on your findings. CTI-VSP asks for a CVSS vector from a description; a MAD of 1.09 to 1.91 on a 0 to 10 scale says nothing about what a finding is worth in your environment.
- Frontier models after mid-2024. The paper's five models date from 2024 and the dataset has been public since June 2024, so a model trained later may have seen it; the only later CTIBench figures are the two small open models above.
- Non-English or broader intelligence work. The authors state the study covers "a limited subset of tasks", is restricted to English, and that questions drawn from public quizzes "may introduce some bias".
How a security team would use it
Use it as a cheap screen on the model behind an agent, before spending time on an agent test. Run your candidate on the released files with the paper's prompt, then compare against the intervals above, not the point values.
| Task | Agent duty it speaks to | Reading the paper's figures |
|---|---|---|
| CTI-MCQ | Vocabulary of the intelligence the agent reads | Highest reported 71.0%: about 29 in 100 questions wrong |
| CTI-RCM | Tagging a finding with its weakness class | Highest reported 72.0%: 280 of 1,000 mapped to the wrong CWE |
| CTI-VSP | A first-pass severity guess to be checked | Closest reported MAD 1.09: a typical miss of over a point on a 0 to 10 scale |
| CTI-ATE | Mapping alert text to ATT&CK techniques | F1 between 0.1562 and 0.6388 |
| CTI-TAA | Suggesting an actor for an analyst to confirm | Correct on 26 of 50 at most; treat output as a lead for review, never as a finding |
If a model falls below ChatGPT-3.5's figures in the paper on the task your agent performs, stop there. If it passes, the next test is your own: build a set from your closed tickets as in evaluating a security agent on your findings, and keep the model choice tied to the success criteria and test sets you set before testing. The same family of questions, with images and SOC data, appears in CyberSecEval 4's CyberSOCEval; the benchmarks index lists the rest, and choosing a model for a security agent sets this screen among the other factors.
Sources
- Alam et al., CTIBench, arXiv 2406.07599 (v3, 11 Nov 2024), read 19 Nov 2026
- CTIBench full text: Sections 3 to 5, Appendix E, Table 1 to 4, read 19 Nov 2026
- CTIBench repository README: dataset sizes and evaluated models, read 19 Nov 2026
- CTIBench project page: results table, NeurIPS 2024 spotlight, read 19 Nov 2026
- Tian et al., CTI-TDS instruction tuning evaluated on CTIBench, arXiv 2610.20284 (14 Oct 2026), read 19 Nov 2026
- Alam et al., AthenaBench, arXiv 2511.01144 (v2, 14 Feb 2026), read 19 Nov 2026
Last reviewed 19 Nov 2026