Conversational AI in customer service has crossed a critical threshold. What began as simple FAQ bots—matching keywords to pre-written answers—has evolved into sophisticated large language model (LLM) powered agents capable of understanding nuance, generating human-like responses, and processing complex multi-step requests. These systems now handle millions of customer interactions daily across every industry, from telecommunications to banking to retail.
The commercial appeal is obvious. AI agents can operate around the clock, handle multiple conversations simultaneously, respond in multiple languages, and scale instantly to meet demand spikes. When they work well, they reduce wait times, resolve issues faster, and free human agents to handle complex cases that genuinely require human judgement.
But there is a fundamental problem. LLMs are probabilistic text generators. They predict the next most likely word given the context, but they have no concept of truth. They cannot distinguish between a correct policy statement and a plausible-sounding fabrication. They do not know the boundaries of their knowledge. And they can generate responses with absolute confidence that are completely wrong—a phenomenon known as hallucination.
When a chatbot hallucinates in a customer service context, the consequences are not academic. The chatbot may promise a refund the company does not offer. It may quote a policy that does not exist. It may provide safety information that is inaccurate. And critically, courts and regulators have made clear that the company is liable for these outputs—the chatbot speaks on behalf of the organisation, and disclaimers do not absolve responsibility.
The Legal Landscape: You Own What Your Chatbot Says
The Air Canada Precedent
The most significant legal precedent for conversational AI liability was established in 2024 when Canada's Civil Resolution Tribunal ruled against Air Canada in a case involving its customer service chatbot. The chatbot had told a passenger that he could book a full-fare flight and then apply for a bereavement discount retroactively. This was incorrect—the airline's actual policy did not allow retroactive bereavement fare applications.
Air Canada argued that the chatbot was a "separate legal entity" and that the passenger should have verified the information on the airline's website. The tribunal rejected both arguments, ruling that Air Canada was responsible for all information provided on its website, regardless of whether it came from a static page or a chatbot. The company was ordered to honour the discount the chatbot had promised.
This ruling established a principle that has since been echoed by regulators globally: a company cannot deploy a customer-facing AI system and then disclaim responsibility for its outputs. The chatbot is an agent of the company, and the company is liable for what it says.
Regulatory Expectations
Beyond case law, regulators are increasingly setting explicit expectations for conversational AI. The EU AI Act requires that AI systems interacting with natural persons must disclose that the consumer is interacting with AI, not a human. The FTC has signalled that deceptive chatbot outputs—including hallucinated information—can constitute unfair or deceptive practices under existing consumer protection law.
The UK Financial Conduct Authority, while not AI-specific, has made clear that firms must ensure customer communications are "clear, fair and not misleading"—a standard that applies equally to AI-generated communications. For regulated industries such as financial services, healthcare, and telecommunications, the standards are even more exacting.
Why Hallucinations Persist
Understanding why conversational AI hallucinates is essential to understanding why pure technical fixes are insufficient.
The Fundamental Architecture
LLMs generate text by predicting the statistically most likely next token given the preceding context. They have no internal model of truth, no access to verified facts at generation time (unless augmented), and no ability to recognise when they are operating beyond the boundaries of their knowledge. This means hallucination is not a bug—it is an inherent property of the architecture.
Training on more data, using larger models, and applying reinforcement learning from human feedback (RLHF) can reduce the frequency of hallucinations, but they cannot eliminate them entirely. Any system that generates text probabilistically will occasionally generate text that is fluent, confident, and wrong.
Data Quality and Knowledge Management
Even with retrieval-augmented generation (RAG)—which grounds the model's responses in a knowledge base rather than relying solely on its training data—hallucinations persist. Common causes include stale or contradictory knowledge base content, poor document chunking that fragments context across retrieval boundaries, information overload from feeding too much context simultaneously, and conflicts between knowledge base content and the model's training data.
RAG reduces hallucinations significantly, but it does not eliminate them. A RAG system with a poorly maintained knowledge base can hallucinate just as confidently as an ungrounded model—the only difference is that the hallucination may be based on outdated information rather than pure fabrication.
Prompt Vulnerabilities
Conversational AI systems can be manipulated through adversarial prompting—carefully crafted inputs designed to bypass safety guardrails. Techniques such as prompt injection, jailbreaking, and role-playing exploits can cause chatbots to ignore their instructions, reveal system prompts, generate offensive content, or provide information they were explicitly designed to withhold.
In a customer service context, prompt manipulation could lead a chatbot to offer unauthorised discounts, reveal internal pricing strategies, bypass verification procedures, or provide information about other customers. These are not theoretical risks—they have been demonstrated in real-world deployments.
Building an Assurance Architecture
Effective assurance for conversational AI requires multiple layers of defence, not a single silver bullet.
The "Escalate, Don't Guess" Philosophy
The most important design principle for customer service AI is: when in doubt, escalate to a human. This sounds obvious, but it runs directly counter to the economic incentive driving chatbot deployment, which is to resolve as many interactions as possible without human involvement.
Effective escalation requires confidence scoring—measuring how certain the model is about its response—combined with intent classification that identifies high-risk scenarios (billing disputes, refunds, safety questions, complaints) and routes them to human agents regardless of the model's confidence level.
Quality Control Gates
Advanced conversational AI architectures implement automated quality control (QC) gates between the model's generation and the customer-facing output. These gates evaluate the response for factual accuracy (cross-referencing against the knowledge base), policy compliance (checking that the response does not promise anything the company cannot deliver), tone and brand consistency, and personally identifiable information leakage.
If a response fails any QC gate, it is either blocked, modified, or routed to a human agent for review. This approach adds latency but dramatically reduces the risk of harmful outputs reaching customers.
Deterministic Actions for High-Stakes Operations
For high-stakes operations like processing refunds, checking order statuses, modifying account settings, or providing billing information, organisations are increasingly replacing probabilistic text generation with deterministic API calls. Instead of having the model "talk its way through" a refund by generating a response about the refund process, the system makes a verified API call to the order management system, retrieves the actual data, and presents it in a structured format.
This hybrid approach—using LLMs for conversational flow and understanding, but deterministic systems for actions and data retrieval—combines the flexibility of natural language with the reliability of traditional software.
Testing and Red-Teaming
Pre-deployment testing for conversational AI must go beyond functional testing. It should include golden set testing (curated question-answer pairs with known correct answers, tested regularly for regression), adversarial red-teaming (dedicated teams attempting to exploit the system through prompt injection, edge cases, and social engineering), bias testing (measuring response quality and accuracy across different languages, dialects, cultural contexts, and user demographics), and stress testing (evaluating performance under high load, with ambiguous queries, and with deliberately challenging inputs).
Post-Deployment Monitoring
Even with comprehensive pre-deployment testing, ongoing monitoring is essential. Key metrics include escalation rates (are they increasing, suggesting the model is encountering more situations it cannot handle?), customer satisfaction scores for AI-handled interactions, conversation abandonment rates, response accuracy measured through periodic sampling, and sentiment analysis across demographic segments.
Monitoring should also track for model drift—changes in the model's behaviour over time due to updates, knowledge base changes, or shifts in the types of queries it receives.
The Bottom Line
Conversational AI in customer service is not going away—the economic incentives are too strong. But the era of deploying chatbots with minimal safeguards and hoping for the best is over. Courts have established that companies are liable for their chatbots' statements. Regulators are setting explicit expectations for transparency and accuracy. And consumers are increasingly sophisticated in their expectations of AI interactions.
The organisations that will deploy conversational AI successfully are those that treat it not as a cost-reduction tool but as a customer-facing representative that requires the same assurance, oversight, and accountability as any human agent—plus additional safeguards to address the unique risks of probabilistic text generation.
Building these safeguards is not optional. It is the cost of deploying conversational AI responsibly, and the alternative—unchecked deployment—carries risks that far outweigh the savings.