The worry about AI automation is almost always the same: the system will send a customer something that is wrong. That worry is fair, and it can be solved. The answer is not to automate less, but to decide in advance who decides what. This article is about that setup: who checks what, when and how. How to check the content of an answer itself is covered in checking AI output.
Set the level of oversight for each step
A process consists of steps: a question comes in, is understood, something is looked up, an answer or action goes out. For each step you choose one of three levels.
- Automatic: routine work without risk, such as answering a question about opening hours or creating a task.
- Automatic with notification: the system acts and a member of staff sees it afterwards, such as an appointment reminder or a booked appointment.
- Approval in advance: anything that costs money or makes a promise, such as a quote, a discount or a cancellation outside the terms.
Write this down before you switch anything on. A short overview per process is enough, as long as your team knows where the line is.
Thresholds: when must a person look?
A threshold is a fixed rule that automatically puts a case in front of a member of staff. Good thresholds are concrete and verifiable, so there is no debate about whether they apply.
- Amount: a quote or order above an amount you choose.
- Subject: a complaint, a damage report or a question about an invoice.
- Doubt: the assistant cannot link the question to information in your own data.
- Customer: a regular business customer or someone with an open issue.
A restaurant might decide, for example, that bookings up to a certain number of guests are confirmed automatically and larger groups go to the manager first. A hotel might always require approval for late check-out requests.
Approval without delay
Approval in advance only works if it is quick. A quote that is drafted automatically but waits three days for sign-off achieves little.
How to keep it moving
- One place: everything awaiting approval sits in one list, not spread across email and chat.
- Ready to send: the member of staff sees the proposal and its source, and approves or edits it in one action.
- A fixed owner: for each type of approval it is clear who does it and who covers during absence.
- A signal: if something waits too long, the customer is told that someone is looking at it.
Spot checks: oversight afterwards
What happens automatically you check afterwards with spot checks. They find mistakes that do not hit a threshold, such as an answer that is slightly off or a tone that does not suit your business.
- Choose a fixed rhythm: in the first weeks a handful of conversations every day, later weekly.
- Vary the type: include ordinary questions, bookings and conversations that were passed on.
- Record what you find: fine, minor error or real error, with a short note on why.
- Agree a limit: after a real error, move that type of question back to approval in advance for a while.
That way oversight grows with the system: what goes right for weeks can move down a level. What goes wrong moves up a level.
A log you will need later
If a customer calls about an answer from last week, you want to see what happened. A log records for each conversation what the question was, which answer or action followed, what information it was based on and whether a member of staff reviewed or changed anything.
The log is also your source for spot checks and evaluations. Do not keep it longer than necessary and include it in your privacy notice, because it contains personal data. The GDPR requires that you keep data no longer than needed for its purpose.
The fallback route to a member of staff
Every AI assistant needs a way out. If it cannot handle a question, or the customer asks for a person, the conversation must reach a member of staff without a detour.
- Recognisable: the customer hears or reads that a colleague is taking over and when they will get an answer.
- Complete: the member of staff receives the summary and the full conversation, so the customer does not have to repeat anything.
- During outages too: if the assistant is not working, calls and messages go to your usual number or inbox.
The AI phone assistant in our software transfers urgent calls or notes a callback request, and the AI assistant passes cases it is unsure about to a member of staff with the summary attached.