Section 1
The difference between fluent and correct
A language model produces the most plausible continuation of a conversation. Plausible is not true, and a customer cannot tell from the phrasing, because a wrong answer arrives in the same confident tone as a right one. The fix is not a better model. It is grounding: the system retrieves the relevant passage from your own documentation, your own pricing, your own policy, and answers from that text rather than from memory. When no passage is found, it says so and hands over. This single design choice separates deployments that hold up from deployments that get quietly switched off. It also imposes a requirement most companies dislike hearing: your policies have to actually be written down, and they have to be current.
Section 2
What it can safely be allowed to do
Draw three rings. The inner ring is read-only: answer questions, look up an order, explain a policy, check a delivery date. Safe to automate broadly. The middle ring is reversible action: book a call, change an appointment, resend an invoice, update a preference. Automate with a log and a confirmation to the customer. The outer ring is irreversible or financial: refunds above a threshold, cancellations, credit decisions, anything that creates a legal commitment. This ring needs a human, or a hard rule the model cannot argue its way past. The rings, not the model, define your risk. Most incidents happen because someone gave an assistant an integration key that spanned all three. The support-specific version of this design is in [Streamlining Customer Service with AI-Powered Chatbots](/blog/streamlining-customer-service-with-ai-powered-chatbots).
Section 3
You are bound by what it says
Treat the assistant's output as a statement by the company, because that is how a customer, and quite possibly a regulator, will treat it. An assistant that invents a discount, a warranty term, or a refund window has made a promise someone will have to honour or publicly withdraw. The practical protections are dull and effective. Restrict answers to retrieved source text. Keep a full transcript log with timestamps. Publish the policies the assistant cites so the customer sees the same version. Review a sample of conversations weekly, chosen from escalations and from very short sessions, since both indicate failure. Disclose that it is an assistant at the start. Customers rarely mind. They mind being deceived about it.
Section 4
Design the handoff before the conversation
Most published complaints about conversational systems are complaints about being trapped. The conversation itself was fine until the customer needed out. So build the exit first, and build it well. A visible route to a person on every turn. Automatic escalation after two failed attempts on the same question. Automatic escalation on detected frustration or on any mention of cancellation, legal action, or a vulnerable circumstance. The handoff must carry context. Nothing destroys the goodwill from a good automated conversation faster than a human agent opening with how can I help, after the customer has already explained the problem twice.
Section 5
Rolling it out without a public incident
Start where the cost of a mistake is lowest and the volume is highest: the twenty questions that make up most of your inbound. Answer those, hand off everything else, and resist the pressure to widen coverage in month one. Run it internally first, with your own support team asking it real historical questions. They will find the failures faster than customers will, and no reputational cost is incurred while they do. Then expose it to a share of traffic, not all of it. Keep the previous path alive and compare outcomes, not opinions. Widen the scope only when the escalation rate for a given question type has been stable for a month. Good transcripts have a second use, as raw material for proof, which [The Power of Customer Testimonials as Stories](/blog/the-power-of-customer-testimonials-as-stories) explores.
Section 6
Containment is a trap as a headline metric
Containment, the share of conversations resolved without a human, is the number vendors lead with. Optimized alone, it rewards a system that makes leaving difficult. Pair it with the metrics that catch that behaviour: repeat contact rate within seven days, satisfaction measured on contained conversations specifically, escalation rate by question type, and abandonment mid-conversation. A conversation that ended without a human and produced a call the next morning was not resolved. It was deferred. On the revenue side, watch conversion on assisted sessions against unassisted, and first response time overall. The strongest signal that a deployment is working is unglamorous: the queue for human agents gets shorter while the satisfaction of the people in it goes up, because the humans are now spending their time on problems that needed a human. Getting the underlying data right first is the argument in [Leveraging AI Automation for Predictive Sales Analytics](/blog/leveraging-ai-automation-for-predictive-sales-analytics).