AI Agent Evaluation
Evaluate whether the tool-using agents you deliver complete tasks, take sensible intermediate steps, behave predictably when things go wrong and stay efficient enough to run in production.
The Challenge
An agent can produce a convincing final response while selecting the wrong tool, passing unsafe arguments, skipping a required confirmation or claiming success when no action ever completed. Output-only testing misses all of it.
We work with software companies, AI consultancies and system integrators as their specialist evaluation partner — white-label or alongside your delivery team — turning agent workflows into reproducible evaluations and reusable quality tests your engineers can rerun after every change.
Four Levels of Agent Evaluation
We assess an agent on four distinct levels, because an agent that reaches the right answer the wrong way is not production ready.
Outcome evaluation
Did the agent actually complete the task correctly? We verify the end state in the systems the agent touched, not just the wording of its final message.
Trajectory evaluation
Were the intermediate steps logical? We review the sequence of decisions, tool selections and arguments to check the agent reached the result through a defensible path.
Behaviour and robustness
Does the agent stay safe and predictable when something goes wrong? We test failing tools, missing data, ambiguous instructions, permission boundaries and required confirmations.
Efficiency
What did the task cost? We measure model calls, tool calls, tokens, wall-clock time and cost per completed task, so performance degradation after system changes becomes visible before your client notices them.
Downstream State Verification
We compare what the agent says happened with what actually happened in the connected system, because a confident final message is not proof that anything changed.
What we verify against
- The CRM record that should have been created or updated
- The database row that should have changed
- The appointment that should appear on the calendar
- The message or email that should have been sent
- The ticket that should have been opened or resolved
- The ERP or billing transaction that should have posted
- The API response and the resulting system state
Metrics we report
- Task Success Rate — verified end-to-end completion, not the agent's self-report.
- Tool Selection Accuracy — how often the agent chooses the correct tool for the step.
- Argument Correctness — how often tool calls use complete, valid and authorised parameters.
- Execution Success — how often invoked tools actually complete without error.
- False-Success Rate — how often the agent reports success the connected system does not confirm.
- State Reconciliation Rate — how often the final system state matches the intended state.
- Recovery Rate — how often the agent recovers or escalates correctly after an error.
- Efficiency — steps, latency and cost per successfully completed task.
What Can Go Wrong
Agent failure modes we evaluate
Wrong Tool Selection
The agent chooses an inappropriate tool, or uses a tool when no action is needed.
Incorrect Arguments
The right tool is called with incomplete, fabricated or unauthorised parameters.
False Success
The agent tells the user an action succeeded although the tool failed, returned no confirmation or was never invoked.
Authority Violations
The agent exceeds the permissions, approvals or business rules defined for the workflow.
Poor Recovery
The workflow loops, silently abandons the task or fails to escalate after an error.
State and Memory Failures
Incorrect or manipulated state changes later decisions and tool use.
Handoff Failures
Required escalation or transfer does not occur, happens too early or loses critical context.
Runaway Cost
Retries, redundant tool calls or context growth make a task far slower and more expensive than intended.
Our Testing Methodology
How we evaluate multi-step, tool-using systems
Scope
We agree the intended use, the agent architecture, the tools and permissions in play, the realistic failure conditions and the acceptance criteria.
Design
We design representative and edge-case task scenarios, the datasets behind them and the rules by which each run is judged.
Calibrate
We decide which checks can be automated deterministically and which need human judgement, then calibrate the graders against reviewed examples.
Evaluate
We run the scenarios reproducibly, capturing trajectories, tool calls, arguments, state changes, outcomes and efficiency metrics.
Verify
We confirm findings manually and convert each confirmed failure into a reusable test that verifies the problem does not return.
What You Get
Agent Evaluation Plan
Documented goals, tools, permissions, scenarios and acceptance criteria.
Scenario Dataset
Representative and edge-case task scenarios you keep and extend.
Failure Evidence
Outcome, trajectory, behaviour and efficiency findings with traces and severity classification.
Reusable Quality Test Suite
Tests your team reruns after each agent release to detect quality loss after system changes.
Evaluate the Agents You Deliver
Start with one agent workflow and a short pilot, delivered white-label or alongside your team.
Discuss an Agent Evaluation Pilot