What EVA measures.
EVA is ServiceNow's open benchmark for enterprise voice agents (code, paper), and its public leaderboard covers the major voice systems from Google, OpenAI, xAI and ElevenLabs. It is built to be hard. Simulated customers call in with real goals and push back the way real customers do, and when the call ends EVA checks what the agent actually changed in the system against what the policy allowed. Across 213 scenarios in airline, IT support and medical HR, each call earns two scores: one for getting it right, one for how the conversation felt.
Simulated caller
Plays a customer with a goal, facts and must-haves, word for word from the scenario file.
Voice agent = model + harness
Hears the caller in real time and decides what to say and which tool to call.
Tools
15 airline tools, 59 for IT, 48 for HR. Look up a booking, rebook, issue a voucher, open a ticket.
Scenario database
A fresh copy for every call. Whatever the agent changes, stays changed.
Database check
A hash of the final database against EVA's expected one. No model involved.
Judge models
GPT-5.2, Claude Opus 4.6 and Gemini 3 Flash review the transcript and the agent's audio.
Audio timing
Measured from the recording of the call.
A call passes EVA-A when it clears EVA's threshold on each accuracy metric, and EVA-X the same way for experience. A score of 0.75 means 75% of calls passed on the first attempt (pass@1). EVA's metric definitions ↗
Two ways to build the same agent
One prompt
- The whole policy sits in one system prompt.
- All 15 airline tools are open on every turn.
- A tool call goes straight to the backend as the model wrote it.
- The model keeps track of where the call is.
Flow harness
- The call is split into stages, each with one objective.
- Each stage opens only the tools that objective needs.
- Typed code checks every tool call before it reaches the backend.
- Rules that code can check are enforced in code, not in the prompt.
The model is identical in both. The harness decides what the model sees at each moment, what it is allowed to do, and what gets checked before anything changes.
Only the harness changed.
We ran EVA's own one-prompt agent, then the Dino flow harness, on the same model, against the same simulated caller and EVA's own airline backend. Here is one call where the difference is easy to hear, and then what happened across the whole controlled run.
I need a hotel voucher and a ride to the hotel for tonight.
She moved her own flight to tomorrow morning. EVA's policy gives hotel vouchers only when the airline caused the disruption. Scenario file ↗
Accuracy level with the best. Experience is next.
EVA scores every call twice. EVA-A asks whether the agent finished the task, stayed faithful to policy and tools, and said things right. EVA-X asks whether the conversation was smooth, short and well paced. Ranked on accuracy, our run sits level with the top of the board. Switch to the trade-off view to see both scores at once: the harness moves Gemini 3.8 Live well up on accuracy and, for now, left on experience.
EVA-A accuracy · pass@1
EVA-X experience · pass@1
Level with Grok Voice 3.0 (0.74), the best published system on these two domains, and above Gemini 3.8 Live with extended thinking high (0.61). Task completion, the database check with no judge involved, is 0.87 against a best published 0.85.
Airline holds EVA's densest policy rules: fees, fare differences, standby, refunds, and vouchers only for real disruptions. Every published airline row is at 0.69 or below.
The plain model scores 0.88. Our misses split between conversation progression (a repeated line, an extra tool call) and turn timing between two live models. What we are doing about it.
28 hard calls, unedited.
Adversarial callers, policy traps, misheard names, up to three requests in one call. Each is a full simulated phone call with Gemini Live on both ends and live tool calls into EVA's backend. Open one to hear it with its transcript and every tool call, including the ones the harness refused.
Calls that show only a task result were scored by EVA's database check; the judges did not run on them.
What the harness does during a call.
A live voice model is quick and natural, and over a long call it drifts from the rules. The harness leaves the conversation to the model and moves the parts that must be exact into structure and code. This is the airline agent. Pick a mechanism to see where it acts.
Every refusal and correction in these calls
Pulled from the 28 calls above, word for word as the model received them. Each one opens the call at that second.
We built the callers too.
EVA's own simulated caller only works inside ServiceNow's harness, so we built our own. For each domain there is a caller agent that carries EVA's scenario word for word, and a simulation engine that puts it on a live line with our agent, both running on Gemini Live in real time. Every call is recorded on both sides, traced tool call by tool call, and scored by EVA's own code. The same lab can run any Gemini Live agent against Gemini Live callers, at scale.
What is not solved yet.
Today the harness costs smoothness. Two causes account for most misses. Conversation progression: a line said twice, or a tool call the conversation did not need. Turn timing: two live models taking turns over a simulated line, which our engine has to get right as much as the agent does.
Target: EVA-X level with plain Gemini 3.8 Live, with EVA-A held.
Where a harness can add the most.
What 2,000 Gemini Live calls showed us.
Running Gemini Live against Gemini Live at volume surfaces behaviour a single call never shows. Each note says what we measured and how sure we are of the cause. Written for teams building on live speech models.
From a strong panel to a full board run.
Same benchmark, our own setup.
We ran EVA's own scenarios, tool backend and scoring code. Some parts of EVA's setup run only inside EVA's harness, and some details are not published, so we rebuilt those parts. This is every difference we know of.