EVA-Bench by ServiceNow ↗ · airline and IT service desk

Thinking-level accuracy. Without the thinking.

Gemini 3.8 Live, with extended thinking off, runs inside the Vox flow harness. On EVA's airline and IT domains, its accuracy is level with the board leader and above Gemini 3.8 Live with extended thinking. On airline it lands above every published system.

* Measured with EVA's own scenarios, backend, and scoring code, on our own rig. See how our setup differs from EVA's.

Results

Accuracy level with the best. Experience is next.

EVA scores each call on two pass rates. EVA-A asks: did the agent finish the task, stay faithful to policy and tools, and say things right? EVA-X asks: was the conversation smooth, short, and well paced? EVA's metric definitions ↗

Vox harness + Gemini 3.8 LivePublished EVA rows95% interval

EVA-A accuracy · pass@1

0.25.5.751

EVA-X experience · pass@1

0.25.5.751

Accuracy

Airline

Experience

The harness effect

Same model. Same runtime. Same caller. Different harness.

We ran EVA's own one-prompt agent on our runtime, then our flow agent, against the same scenarios, the same simulated caller, and EVA's own booking backend. The only change is the harness.

Listen

The hardest calls, unedited.

Adversarial callers, policy traps, and three or four requests in one call. Each recording is a full simulated call: a Gemini Live caller against a Gemini Live agent, with live tool calls into EVA's own backend. Tap a call to hear it with its transcript and every tool call.

How it works

The model talks. The harness keeps it exact.

A live voice model is fast but forgets rules over a long call. The Vox flow harness splits the call into steps, gives each step only the tools it needs, and checks every tool call before it reaches a backend.

Counterpart agents and the simulation engine

We build the callers too.

Every call here is agent against agent. The caller is a Vox flow agent with EVA's scenario goals, run on Gemini Live. The engine runs both sides in real time and scores each call with EVA's own code.

Medical HR

Where we have the most room.

What comes next

From strong accuracy to a full board run.

Field notes

What 2,000 Gemini Live calls taught us.

Running Gemini Live against Gemini Live at scale shows behaviour that one call never shows. Each note gives what we measured and how sure we are of the cause.

Our setup next to EVA's

Same benchmark. Our own rig.

We ran EVA's own scenarios, tool backend, and scoring code. Some parts of EVA's rig run only inside EVA's harness, and some details are not published, so we rebuilt those parts. This table lists every difference we know of.

Method

How we measured.

Every number on this page comes from saved calls and EVA's own scoring code. These notes say exactly what was run.

CallerAgent
0:00 / 0:00