Our Methodology
A structured, evidence-based way to evaluate the AI-enabled workflows you deliver — the RAG systems, agents and connected processes behind them — from definition to reusable quality tests.
How We Work
Every engagement follows the same five steps, designed to turn uncertainty into evidence your engineers can act on.
Define
We agree the intended use of the workflow, the outcome it needs to produce, the risks that matter for your client and the acceptance criteria the evaluation will be judged against.
Observe
We map the components of the workflow and decide which evidence to capture at each step — retrieval, model output, tool calls, arguments, database and API responses, the resulting state and human action.
Evaluate
We run representative and boundary cases reproducibly, and repeat them where stochastic behaviour matters, recording inputs, configurations, traces and results so every finding can be re-run.
Isolate
When something fails, we identify where it originates — retrieval, prompt or model, routing, tool selection, arguments, permissions, API or database, workflow state or handoff — instead of stopping at the symptom.
Verify
We rerun the evaluation after each change and convert confirmed failures into reusable tests, so your team can detect quality loss after every future system change.
Three Evaluation Dimensions
Each system in scope is evaluated across three complementary dimensions.
Retrieval and grounding
Does the system find the right evidence and stay supported by it? We evaluate retrieval quality, completeness of context, citation correctness and correct refusal when the answer is not in the sources.
Agent tasks and tool use
Does the agent complete the task through a defensible path? We evaluate outcomes, trajectories, tool selection and arguments, recovery after errors and efficiency per completed task.
Robustness and failure behaviour
Does the workflow stay reliable under difficult and unexpected conditions? We add scoped scenarios for edge-case inputs, incomplete or manipulated context, unsafe tool requests and sensitive-information exposure.
Evidence Tied to the Version You Actually Tested
An evaluation result only means something when it is clear which exact combination was tested. Every reliability evidence pack we produce records the versions behind the result:
This is not a governance claim. It simply means the evidence should remain reproducible after the system changes — and that a passing result on an old version tells you nothing about the version now in production.
Our Principles
Evidence over opinion
Every finding is backed by reproduction steps and traces. We never report suspected issues without evidence.
Risk-proportionate testing
We allocate testing depth based on failure impact — high-risk areas get more rigorous evaluation.
Reproducibility
Our results can be independently verified. We document inputs, configurations and expected outcomes.
Honest about limitations
We clearly distinguish between what we tested and what lies outside the scope. No conclusion extends beyond our actual evaluation.
Apply this methodology to the systems you deliver
Start with one RAG system or agent workflow — white-label or alongside your delivery team.
Discuss a Partner Evaluation