Most claims about AI in the SOC rest on one number: how often the agent gets the verdict right. It’s the wrong measure. An agent can land the correct verdict and still leave the real questions unanswered: how the attacker got in, what they reached, whether they appeared elsewhere. Defenders act on those answers, not on the label.
With Vega II, we described our agentic approach: frontier models, optimized and orchestrated by the platform, with the Vega Model (proprietary, based on open weights) alongside them. That design has to earn trust with evidence. The Vega Bench Series is how we show the work, and it sets the standard every agentic system should be held to. It starts with investigation, and each benchmark that follows uses the same scorecard.
Why a correct verdict isn’t enough
An illustrative case: an agent sees a sign-in from an unfamiliar hosting-provider IP and labels it malicious. Correct. But it never establishes how MFA was bypassed, what the session accessed, or whether the same infrastructure touched other accounts. The verdict matches. The response depends on everything the agent skipped.
Verdict accuracy has two further problems. Most incidents that reach investigation are benign, so an agent biased toward “benign” scores well while failing the cases that matter. And two models with the same verdict accuracy can investigate very differently: one checks peer hosts and the user’s baseline, the other reasons from the alert alone. A single pass or fail can’t tell them apart.
One scorecard, four axes
Measuring agentic cyber defense takes four axes, and the series scores all of them in every benchmark. We report each one separately because they trade against each other. Faster can mean less investigation. More accurate can mean far higher spend.
| Axis | What it asks | How VCI-Bench measures it |
|---|---|---|
| Speed | Time to a completed result | End-to-end latency |
| Accuracy | Findings against ground truth | Verdict accuracy, finding coverage, false claims per case |
| Cost efficiency | Compute per case, read alongside accuracy | Model spend per investigation |
| Intelligent data access | Whether the system reaches the data the case requires | Pivot coverage (executed queries by source, entity, and time window). Guardrail use is scored in our model benchmark, and a VCI-Bench metric is planned |
The standard behind the scores
Whatever the task, a benchmark worth trusting meets this bar, and every one in this series does:
- The same environment, tools, and graders for every arm.
- Real difficulty, frozen so every run sees the same evidence.
- Actions graded against what the agent executed, not what it claims.
- Atomic, researcher-approved ground truth.
- Limits published with the numbers.
What the series is for
It has two jobs. The first is a fair comparison: Vega’s agentic stack against frontier models on their own, in the same environment, read per axis and per case category. The second is a basis for decisions about the stack itself: which components earn their place (system prompt, skills, query translation), and where the Vega Model can replace one. Where results support it, that replacement happens component by component, and always opt-in.
First up: VCI-Bench
VCI-Bench (Vega Cyber Investigation Benchmark) evaluates whether an agent can investigate a security incident end to end. Each case gives the agent an incident as production received it and a frozen capture of the relevant logs, which it investigates with the same query engine, schemas, and tools it uses in production.
We don’t compare its report to a reference report, because two correct reports can look nothing alike. Instead, the report and every query the agent ran are graded against a record of atomic, true-or-false criteria: findings the report must establish, searches the agent must run, and false claims it must avoid.
The benchmark uses 264 production incidents as evaluation material, about 3,300 criteria (roughly 1,600 findings, 1,000 searches, and 700 false claims) and more than 30 log sources. Cases span endpoint, identity and access, cloud, data access and sharing, network, email and phishing, and AI and developer activity. The full build is in the VCI-Bench deep dive, which follows this post.