A Correct Verdict Is Not Enough: Introducing VCI-Bench

TL;DR

• VCI-Bench scores how an AI agent investigates: 264 production incidents, about 3,300 binary criteria, and more than 30 log sources.
• Each case carries a researcher-approved record of the findings a report must state, the searches the agent must run, and the false claims it must not make.
• Agents work in frozen captures of production logs, up to 500 million rows per case, that keep the evidence, the comparable peers, and the false leads.
• Code checks verdicts and pivots, three LLM judges grade findings and false claims against quoted report text, and every case runs three times.
• Results come in the next post: the Vega investigation agent against frontier models on the same cases, data, and criteria.

An AI investigation agent can reach the right verdict while leaving important questions unanswered. How did the attacker get in? What did they reach? Was similar activity present elsewhere? A verdict alone tells us little about whether the agent did the work needed to support a response.

At Vega, we built an AI agent that investigates security incidents end to end. VCI-Bench (Vega Cyber Investigation Benchmark) measures the investigation itself, on real production incidents.

We built it to guide improvements to Vega's investigation agent. It checks whether an agent establishes the required findings, performs the necessary searches, and avoids false claims, while also measuring investigation speed and cost.

VCI-Bench opens our series on how agentic cyber defense should be measured. This post covers how it is built and what it scores; experiments and baselines follow in the next one.

Why the Verdict Is Not Enough

Evaluating investigation requires tasks that leave the agent responsible for deciding what to search, which explanations to test, and when the evidence supports a conclusion. Related work such as ExCyTIn-Bench evaluates agents through security questions derived from investigation graphs in a controlled Azure tenant. Our evaluation starts with a production incident and follows the agent through to its verdict and report, checking the findings and investigative steps required for that case.

The most direct measure of an investigation is its verdict: collect incidents with known outcomes and count how often the agent agrees. Verdict accuracy is a useful number, but on its own it has three problems.

Base rate. Most incidents that reach investigation are benign. An agent biased toward “benign” scores well on accuracy while failing on the cases that matter most.

Correct verdicts with incorrect reasoning. An agent can label an account takeover malicious based on an unusual sign-in location without establishing how the attacker bypassed MFA, whether persistence was created, or what was accessed. The verdict matches, but the response depends on the parts the agent never established.

Resolution. Verdict accuracy reduces each run to a single pass or fail. Two models with the same verdict accuracy can investigate very differently: one checks the peer hosts and the user's baseline, the other reaches the same conclusion from the alert alone. That single result cannot separate them or identify the missing work we need to improve.

The environment matters too. An investigation depends on choosing where to look among sources, entities, and time ranges, while distinguishing relevant evidence from routine activity. The evaluation environment has to preserve that difficulty. We describe how in The Environment Problem.

An investigation is a set of conclusions an investigator should reach and checks an investigator should perform. Each of these can be stated as a binary criterion and verified independently. The benchmark is built on that decomposition.

Anatomy of a Case

Each case has four parts: the input, the environment, the output, and the verification.

In our illustrative scenario, a user enters their password and a one-time code generated by an authenticator app (TOTP) on a phishing page. The attacker immediately uses both to sign in to the real service before the code expires. Multi-factor authentication (MFA) succeeds, but it is the attacker who gains access.

Figures 1 and 2 follow this same scenario. Its entities, sizes, query results, and run outcomes are illustrative, not measurements from a real case.

Figure 1: One case, end to end

Input. The incident exactly as production received it: the alerts, each detection's logic and query, and the matched event samples.

Environment. A frozen capture of the relevant production logs, including activity from comparable users and hosts, false leads, and baseline activity.

Output. The agent's final report, including its verdict, findings, and reasoning, together with a record of every query it executed. These let us evaluate both its conclusions and the investigative steps behind them.

Verification. The agent's report is not compared to a reference answer. Two correct reports on the same incident can differ substantially in wording and structure, and similarity metrics reward style. Each case instead carries a structured record of what a correct report must contain and what it must not.

Case selection. Cases are chosen from challenging production incidents: true positives that are easy to miss, benign incidents that look malicious, and incidents where the initial evidence is misleading. Incidents whose answer is fully contained in the alert are excluded.

Breaking an Investigation Into Findings

The record consists of a verdict specification and a list of criteria. Each criterion is a single sentence that is either true or false of the agent's work. Criteria come in three types.

• Findings are facts the report must state, for example “The evidence supports TOTP phishing as the sign-in mechanism.” They measure whether the agent understood what happened.
• Pivots are required investigative searches, for example “Queried at least 14 days of the user's Okta history.” These searches follow leads into other sources, entities, or time ranges and check whether the conclusion is backed by investigation. Pivots are graded against the queries the agent executed, not against what the report claims.
• False claims are incorrect conclusions the report must not make, for example “The suspicious sign-in succeeded because the user approved an MFA push” when the sign-in logs record TOTP instead. They are drawn from hypotheses that investigators or earlier agent runs considered and rejected.

An illustrative investigation

What the agent receives. Four alerts on a single user within twenty minutes: an Okta sign-in from a hosting provider's IP range absent from the user's captured history, an unfamiliar device fingerprint, an impossible-travel match against the user's previous session, and a Salesforce session from the same address. The agent receives the alerts, their detection logic, and matched event samples. It can query an illustrative capture of about 140 million rows across Okta system logs, Salesforce event logs, and web proxy logs.

What it must discover. The agent must establish whether the account was taken over, how MFA was passed, and what the attacker accessed afterward. In this illustration, proxy logs show the user visiting the phishing page immediately before a successful TOTP sign-in from the unfamiliar hosting-provider IP. Correlating those events supports the phishing explanation; TOTP success alone does not.

The misleading lead. Earlier declined Okta Verify push prompts suggest push fatigue, but the successful sign-in used TOTP. The agent must distinguish those rejected prompts from the MFA step that granted access.

The Salesforce session creates another question: what happened after sign-in? In this example, the available logs establish access but do not establish what the attacker did inside Salesforce. A correct report must make that limit clear.

The illustrative record defines the findings to establish, the searches to perform, and the incorrect claims to avoid. Figure 2 shows selected criteria across three hypothetical runs. All three reach the same verdict but differ in what they establish, search, and claim.

Figure 2: One case, broken into findings

The verdict specification lists the expected verdict and the set of accepted verdicts. A benign case where “suspicious” is a defensible conclusion may accept both. A malicious case accepts only “malicious”.

Each case is stored as a single record. This trimmed record uses the same illustrative scenario:

{
  "case_id": "illustrative-account-takeover",
  "verdict": { "expected": "malicious", "accept": ["malicious"] },
  "checks": [
    { "id": "mfa-relayed-totp",
      "check": "The evidence supports TOTP phishing as the sign-in mechanism.",
      "grader": { "type": "judge" } },
    { "id": "looked-sign-in",
      "check": "Queried Okta events for user-042 around the suspicious sign-in.",
      "grader": { "type": "query", "source": ["Okta"],
                  "kql_contains_any": ["user-042"],
                  "span_covers": "2026-09-14T09:12:00Z" } }
  ],
  "false_claims": [
    { "id": "push-fatigue",
      "claim": "The suspicious sign-in succeeded because the user approved an MFA push." }
  ]
}

The Environment Problem

An investigation benchmark has to preserve both the evidence and the difficulty of finding it. Copying an entire production environment is impractical: a single firewall source can produce around 1.8 billion rows a day. Keeping only the events that triggered an alert removes the context needed to interpret them and makes the investigation artificially easy.

We build each capture around the questions an investigator needs to answer. Has this user signed in from this IP before? Did other hosts contact the same domain? What happened before and after the alert? Those questions determine which data must remain complete and where we can reduce volume.

The capture preserves complete detection results, including the underlying events for aggregated detections. It also retains the relevant users' and hosts' activity and every occurrence of the case's indicators across the covered sources and time ranges. Counts are evidence: sampling these slices could turn a recurring sign-in into an apparent first occurrence, or hide another affected account.

Comparable users and hosts are captured at the same fidelity. Otherwise, the suspect could stand out simply because we retained more of its history. We also keep entities the original investigation ruled out, so the agent must work through plausible alternatives rather than follow a trail containing only relevant evidence.

Around the incident timeline, we retain an unfiltered window of surrounding activity, sized to fit the source's volume. Beyond it, we sample broader background and selected adjacent sources where an investigator might look. Smaller sources are captured in full when they fit the budget. Each case has a ceiling of 500 million rows; we stop adding data once the required questions are covered.

Figure 3: How big is the haystack

Complete evidence slices support exact counts within their scope; sampled background provides context. We record coverage limits for each case.

Building a realistic evaluation environment requires more than saving logs: the agent must be able to investigate them. Vega's existing query engine and investigation tools provide that foundation. We reuse the same schemas, query interface, and code execution capabilities used in production, so each frozen capture becomes a working investigation environment. Our work then focuses on preserving the right evidence and defining how to evaluate the investigation.

The agent is not told that the data is frozen. Each case runs in an isolated scope. These conditions make it especially important to check that a missing result reflects the original evidence rather than a gap introduced by capture.

We replay detection and investigation queries to check that the capture preserves the evidence needed for each case.

These evidence checks are separate from comparing the agent's scores. In comparisons of investigations against the original environments and their evaluation captures, we observed matching evaluation metrics. That supports preservation of the measured behavior on the cases tested; it is not proof that every query returns the same result or that every capture has passed all replay checks.

Scoring a Case

A run gives the agent the incident and access to the capture, then collects its final report and executed-query log. Code compares the verdict with the accepted set. A panel of three LLM judges grades findings and false claims against the report, one criterion at a time. We average the three judges' binary scores for each criterion, as in Artificial Analysis' LAB evaluation. Pivots are checked against executed queries for the required source, entity, and time window.

Our report judges see the report and researcher-reviewed criteria, not query results. For each finding a judge credits, it must quote supporting text, and code verifies that the quote appears in the report. This makes decisions traceable without guaranteeing that the judges interpreted the text correctly.

Each case runs three times to examine variation in investigation behavior. Those repeated investigations are separate from the three judges assessing each report. We organize the measures under four axes:

We report these measures separately. Faster or cheaper investigations may leave findings uncovered; higher verdict accuracy may coexist with more false claims. Reading the axes together lets us assess the trade-offs and identify where Vega's components make a difference.

Where the Ground Truth Comes From

Our security researchers built a seed set of evaluation records from production incidents, checking existing AI investigations against the underlying logs. They corrected errors, removed unsupported claims, and added missing findings. These records established the reference standard for the broader authoring pipeline.

Expert-defined criteria also underpin evaluations such as Harvey's Legal Agent Benchmark (LAB).

A single investigation may miss relevant evidence or plausible explanations. Our authoring pipeline combines independent investigations from different models and harnesses, then checks their hypotheses against the captured data before researcher review.

Figure 4: Drafting records at scale

For each frozen case, several strong agents investigate the incident independently, using different models in different harnesses, with far larger turn and cost budgets than production allows. Different models and harnesses fail in different ways, so the union of their hypotheses is broader than any single run. Each hypothesis is then tested against the frozen data until it is confirmed, rejected, or shown to be undeterminable from the logs. Confirmed hypotheses become findings and pivots, rejected ones become false claims, and undeterminable ones are excluded.

This level of effort is not viable in production, where an investigation must finish in minutes at a few dollars. It is viable once per case, and it broadens the set of evidence-backed findings considered for each record.

Controls and Limitations

Security researchers check drafted criteria against the frozen data, resolve disagreements, and approve every record.

We validated the automated graders against independent researcher assessments on a sample of reports. Researchers graded without seeing automated decisions, then reviewed disagreements to identify grader errors, ambiguous criteria, or incorrect expectations. This gives us confidence in the graders on the reports reviewed.

What's in VCI-Bench

The set holds 264 cases and about 3,300 criteria: approximately 1,600 findings, 700 false claims, and 1,000 pivots. The cases span more than 30 log sources, and a single case typically involves between two and eight of them.

Figure 5: Case and criteria distribution

Cases are grouped by primary investigation focus: endpoint activity, identity and access, cloud infrastructure, data access and sharing, network activity, email and phishing, and AI and developer activity. Figure 5's illustrative allocation is informed by a review of recent incidents across tenants. The verdict mix deliberately oversamples malicious incidents to about a quarter of the set. Benign cases remain the majority because correctly closing benign incidents, with supported reasoning, is a substantial part of an investigator's work.

What's Next

Next, we'll compare Vega's investigation agent with frontier models on the same cases, frozen data, and grading criteria, then isolate the contribution of our investigation instructions, skills, and query translation.

Where results support it, we will evaluate the Vega Model, a proprietary model built on open weights, on individual components, category by category, with frontier models working alongside it. Any such change would be opt-in. We will also examine whether a cyber-focused Vega Model can reduce unnecessary refusals on authorized defensive work compared with general-purpose frontier models.

The case set will grow alongside that work, adding underrepresented incident types, more environments, and findings confirmed through researcher review. VCI-Bench gives us a common basis for testing changes and showing which parts of an investigation they improve.