Insights
Practical articles on evaluating RAG systems and tool-using AI agents
Trace-Based Debugging: Root-Causing an AI Workflow Failure
When an AI workflow produces a wrong output or takes a wrong action, the cause could be anywhere in the chain: retrieval, context assembly, generation, action selection or downstream execution. Trace-based debugging makes each step visible so you can find the root cause instead of guessing.
Release Gates for AI Workflows: The Evidence You Need Before Shipping
Traditional software has clear release criteria: tests pass, code reviewed, staging verified. AI workflows need different release gates — ones that account for probabilistic behaviour, evaluation drift and the gap between average quality and worst-case failure.
Correct Escalation and Abstention: When an AI Should Say "I Don't Know"
The most valuable thing an AI workflow can do is recognise when it should not act. Correct abstention and escalation prevent the highest-severity failures — yet most evaluation frameworks ignore them entirely.
Regression Testing for AI: Catching Silent Degradation After a Prompt or Model Change
A prompt tweak, a model upgrade, a knowledge base refresh — each can silently change the behaviour of an AI workflow in ways that standard monitoring misses. Regression testing for AI requires a different approach than regression testing for traditional software.
Building a Behaviour Contract for an AI Workflow
Before you can evaluate an AI workflow, you need to define what correct behaviour looks like. A behaviour contract makes expectations explicit, testable and shared between the team that builds the system and the team that evaluates it.
Testing AI Agents That Touch Your CRM, API and Database
When an AI agent can update your CRM, call external APIs and write to your database, a wrong answer becomes a wrong action with real consequences. This guide covers the evaluation approach for agents with downstream system access — from state verification to rollback testing.
Evaluating RAG When Retrieval Feeds a Downstream Action
Most RAG evaluation stops at answer quality. But when retrieved context feeds an agent that updates a database, sends an email or creates a ticket, retrieval failures cascade into operational damage. Here is how to evaluate RAG systems that do more than answer questions.
From Demo to Production: Why AI-Enabled Workflows Break at the Seams
The demo worked perfectly. The production deployment fails silently. The gap between a convincing AI demonstration and a reliable production workflow is where most projects stall — and where systematic evaluation makes the difference.
AI Vendor Due Diligence: A Practical Evaluation Framework
AI procurement demands due diligence across five risk dimensions: technical, operational, financial, legal, and ethical. A structured evaluation framework for organisations deploying third-party AI systems.
Model Drift: The Silent Threat to AI Systems in Production
Ninety-one percent of production AI models experience drift. Without proactive monitoring, error rates can increase by 35 percent within six months. A comprehensive guide to detecting and managing model degradation.
Agentic AI: Why Autonomous Systems Need a New Assurance Playbook
Agentic AI systems can plan, reason, and act autonomously — but their failure modes cascade in ways traditional testing cannot catch. A practical guide to assurance for autonomous AI.
When AI Agents Speak for Your Organisation: Evaluating Customer-Facing Workflows
AI agents now handle customer conversations, process requests and take actions on behalf of organisations. Courts have ruled companies liable for what these agents say. This analysis examines where customer-facing AI workflows fail — hallucinated promises, policy violations, escalation failures — and the evaluation architecture needed to catch these failures before they reach production.
Measuring AI Hallucinations: From Anecdote to Metric
Everyone knows AI systems make things up. Far fewer organisations can say how often, how badly, or whether last month's fix actually helped. Turning hallucination from a scary story into a number you track is what separates managed risk from wishful thinking.
Why RAG Systems Fail — and How to Evaluate Them Properly
Retrieval-augmented generation is the most popular way to make a language model answer from your own documents. It is also quietly one of the easiest architectures to get wrong. Here is where these systems break, and how to test them honestly.
What AI Assurance Actually Means — and Why Every Sector Needs It
AI assurance is not a certificate or a one-off audit. It is the ongoing discipline of proving that an AI system does what it claims, keeps doing it after deployment, and fails safely when it does not. Here is what that looks like in practice.