Retrieval-augmented generation has become the default architecture for grounding language models in organisational knowledge. The standard evaluation approach measures whether the system retrieves relevant documents and generates accurate, grounded answers.
This works when RAG is used as a question-answering system. But increasingly, RAG is a component in a larger workflow where the retrieved context informs a decision or action: an agent reads a policy document and decides whether to approve a refund; a support system retrieves a troubleshooting guide and creates a ticket with specific diagnostic steps; a compliance workflow retrieves regulatory requirements and flags non-conforming records.
When retrieval feeds a downstream action, the evaluation requirements change fundamentally. A retrieval failure that produces a slightly wrong answer is an inconvenience. A retrieval failure that triggers the wrong business action is operational damage.
The Action Chain
In an action-oriented RAG workflow, the chain is: query → retrieval → context assembly → generation → action decision → API call → state change.
Each link in this chain can fail independently, and failures compound. Poor retrieval feeds bad context to the model. Bad context produces a wrong interpretation. A wrong interpretation triggers an incorrect action. The incorrect action changes state in a production system — a database record updated, a customer email sent, a ticket escalated to the wrong team.
The critical difference from question-answering RAG is that there is no human reading the answer and applying judgement before anything happens. In a chatbot, a user can notice a strange answer and ignore it. In an action chain, the wrong interpretation becomes an irreversible state change before anyone reviews it. The evaluation must therefore stand in for the human judgement that has been removed from the loop.
Evaluating Retrieval for Action Contexts
Precision Matters More Than Recall
For question-answering RAG, recall is usually the priority — you want to make sure the relevant information is retrieved, even if some irrelevant passages come along. For action-oriented RAG, precision matters more. An irrelevant retrieved passage that contradicts the relevant one can flip the model's action decision.
Measure retrieval precision not just as "relevant vs irrelevant" but as "action-safe vs action-dangerous." A retrieved passage is action-dangerous if it could plausibly cause the model to take a different action than it would with the correct context alone.
Contradiction Detection
When multiple retrieved passages contain contradictory information — an old policy and a new policy, a general rule and a specific exception — the model must handle the contradiction correctly. In a question-answering context, it might acknowledge the ambiguity. In an action context, it must either choose the correct interpretation or escalate.
Test explicitly for contradiction handling: retrieve passages with known contradictions and verify that the model either resolves them correctly or refuses to act.
Freshness as a Safety Property
Stale retrieved content is a quality issue for question-answering and a safety issue for action-oriented workflows. If the model retrieves an old refund policy and processes a refund that the current policy does not allow, the damage is concrete and measurable.
Evaluate freshness not as a nice-to-have metric but as a safety property. Measure the age distribution of retrieved passages for action-triggering queries and set maximum staleness thresholds for different action categories.
Evaluating the Action Decision
Decision Boundary Testing
The model makes an implicit decision: given this context, what action should I take? Test the decision boundary explicitly by varying the retrieved context and measuring when the action changes.
If a small change in retrieved context — removing one sentence, adding one contradictory passage — flips the action decision, the workflow is fragile. Robust workflows produce stable action decisions across reasonable variations in retrieved context.
Escalation as a First-Class Action
For action-oriented RAG, "I am not confident enough to act" is the most important possible output. Test whether the model escalates when the retrieved context is ambiguous, contradictory, insufficient or outside the scope of its authority.
Missed escalations — cases where the model should have deferred to a human but instead took an action — are the highest-severity failures in action-oriented RAG.
A Worked Example: The Refund Agent
Consider a workflow that reads a customer's order history and the refund policy, then decides whether to issue a refund automatically. It looks like a clean RAG problem: retrieve the policy, retrieve the order, generate a decision.
The failures are all at the seams. If retrieval returns last year's refund policy, the agent approves a refund the current policy forbids — a freshness failure. If it retrieves both the general 30-day policy and the specific exception for final-sale items but the general passage ranks higher, the agent may miss the exception — a contradiction-handling failure. If the order history is incomplete because one system was slow to sync, the agent decides on partial data and never signals that it is missing information — a missed escalation.
Evaluating this workflow means constructing test cases for each of these failure modes deliberately: an outdated-policy case, a general-versus-exception case, an incomplete-data case. For each, the acceptance criterion is not "did it produce a fluent explanation" but "did the correct state result — the right refund amount, or a correct escalation to a human."
End-to-End Evaluation
State Verification
The ultimate measure of an action-oriented RAG workflow is whether the resulting state is correct. After the workflow processes a query, check the downstream system: was the right record updated? Was the right email sent? Was the ticket created with the correct fields?
This requires evaluation infrastructure that can inspect downstream system state — database queries, API response verification, message queue inspection. It is more complex than checking whether an answer is correct, but it is the only way to verify that the workflow actually works.
Regression Testing Across the Full Chain
A knowledge base update can change retrieval results, which changes generated context, which changes action decisions. Test the full chain after every change, not just the component that changed.
The most dangerous regressions are those where retrieval quality stays the same by standard metrics but the action outcomes change because the model interprets slightly different context differently.
The Practical Takeaway
If your RAG system only answers questions, evaluate it like a question-answering system. If your RAG system feeds actions — if it updates databases, sends communications, creates records, or triggers workflows — evaluate it like a production system that makes decisions on behalf of your organisation. Because that is what it is.