Human-in-the-Loop: How to Use AI in Workflows Without Losing Control

The most common worry we hear about AI automation is not "will it work?" but "what happens when it gets something wrong?" It is the right worry. Language models are very capable and occasionally wrong in ways that are hard to predict. If an automation sends a wrong price to a customer, approves a refund it should not, or overwrites good CRM data with bad, the cost of cleaning up can outweigh the time saved.

Human-in-the-loop design is the answer. It means deciding, step by step, where people review, approve or correct what the AI does, and building the workflow around those points. Done well, it lets you get value from AI from the first week while keeping people in charge of the decisions that matter. This guide covers the patterns, how to set thresholds, and what good review tooling looks like.

Why "fully automatic" is the wrong starting goal

Teams often picture success as a process with no people in it. In practice, the best AI workflows have people in exactly the right places. A few reasons:

• Models fail unevenly. They can handle hundreds of routine cases perfectly, then misread an unusual one with total confidence. Reviews catch the unusual ones.
• Accountability stays with people. Your customers, auditors and regulators will hold a person responsible, not a model. Approval steps make that explicit.
• Corrections are training data. Every time a reviewer fixes something, you learn where the system is weak. Without review, those lessons are lost.
• Trust is earned. Teams adopt automation faster when they can see and correct what it does before it acts.

The goal is not to remove people. It is to move them from doing repetitive work to checking exceptions and making judgment calls.

Five patterns for keeping humans in the loop

1. Draft and approve

The AI prepares something and a person approves, edits or rejects it before it goes anywhere. Drafted customer replies in Zendesk or Gorgias, proposed CRM updates after a sales call, suggested GL coding for an invoice.

Use when: output is visible to customers or changes important records, and volume allows a person to glance at each item. This is the safest starting point for almost any AI workflow.

2. Confidence-based routing

Items the system is sure about go straight through. Items it is unsure about go to a review queue. "Sure" should be defined by checks you can verify, not only by asking the model how confident it feels. For example: extracted totals add up, the customer exists, the SKU matches, and the classification agrees with a simple keyword rule.

Use when: volume is high and most cases are routine. This is the pattern behind most document processing pipelines.

3. Approval gates on actions

The AI can investigate and recommend freely, but specific actions require a person to click approve. An agent might research a refund request, check order history and propose "refund $45 to original payment method", then wait for a support lead to approve in Slack.

Use when: you are using an AI support agent or other agent that can take actions involving money, customer commitments or data deletion.

4. Limits and auto-escalation

The AI may act on its own within defined limits, and anything outside them escalates. Refunds under a set amount, discounts up to a set percentage, replies only on certain ticket categories, no more than a set number of actions per customer per day.

Use when: you have run the workflow with approvals for a while, the error rate in a category is consistently low, and the cost of a mistake within the limit is acceptable.

5. Sample review after the fact

The AI acts on its own, and a person reviews a random sample afterward, plus anything flagged by customers or monitoring. Think of it as quality assurance rather than approval.

Use when: volume is too high for per-item review, mistakes are cheap and reversible, and you have good logging.

Choosing the right pattern for each step

Most workflows mix patterns. Use two questions to decide for each step: how costly is a mistake, and how easy is it to reverse?

	Easy to reverse	Hard to reverse
Low cost if wrong	Act automatically, sample review (tagging tickets, internal summaries)	Confidence routing (CRM field updates, internal notifications)
High cost if wrong	Limits and auto-escalation (small refunds, draft orders)	Draft and approve, or approval gates (payments, customer contracts, deletions)

Revisit the table every few weeks. As you gather evidence, steps can move toward more autonomy. Some, like changing supplier bank details or signing contracts, should probably never move.

A worked example: AI in a support workflow

Take a hypothetical ecommerce brand with a small support team using Gorgias. They want AI to help with a high volume of order status, return and product questions.

Weeks 1 to 4: draft and approve. For every ticket, the AI classifies it, looks up the order, checks tracking, and drafts a reply with any proposed action. Agents review every draft. The team tracks how often drafts are sent unchanged, lightly edited or rewritten, broken down by ticket type.

Weeks 5 to 8: confidence routing for one category. "Where is my order?" drafts are almost always sent unchanged, and the data comes straight from the carrier API. Those replies now send automatically when tracking data is fresh and the order has no open issues. Everything else is still reviewed.

Week 9 onward: limits. The AI can approve return labels for orders within the return window that meet the policy, and offer store credit up to a small fixed amount for minor delivery delays. Larger amounts, damaged items and anything involving a complaint still go to a person. A team lead reviews a sample of automated replies each week.

The team never had to bet on the AI being right on day one. Each step toward autonomy was backed by data from the previous one.

What a good review experience looks like

Human-in-the-loop only works if reviewing is fast. If approving a draft takes as long as writing it from scratch, people will rubber-stamp or bypass the system. A good review screen:

• Shows the source material (email, document, ticket) next to the AI's output
• Highlights why the item needs review: which check failed or which field is uncertain
• Lets the reviewer edit inline and approve in one click
• Supports keyboard shortcuts and batch approval for similar items
• Captures a reason when the reviewer rejects or heavily edits
• Lives where the team already works: inside the helpdesk, in Slack or Teams, or in a dedicated queue

For simple approvals, a Slack or Teams message with approve and reject buttons is often enough. For anything with several fields to check, a purpose-built internal tool usually pays for itself quickly.

Logging and audit trails

Every AI-assisted decision should leave a record you can look up later:

• The input the AI saw (or a reference to it)
• What it produced or proposed, including which model and prompt version
• Which checks ran and their results
• Who approved, edited or rejected it, when, and what they changed
• What action was finally taken in which system, with record IDs

This record answers the question "why did this happen?" when a customer complains, and it gives you the data to decide when a step is ready for more autonomy. It also matters for compliance in regulated industries and for financial processes that auditors will look at.

Guardrails beyond human review

People are one layer of protection. Others are just as important:

• Least privilege. Give the AI access only to the tools and data a step needs. An agent that drafts replies does not need permission to delete customers.
• Validation in code. Check formats, totals, allowed values and business rules with ordinary code before anything is written to a system.
• Rate and spend limits. Cap how many actions or how much value the automation can move per hour or day, and alert when limits are approached.
• Kill switch. Make it easy for an ops lead to pause automation and fall back to manual handling.
• Monitoring. Alert on sudden changes in volume, error rates or the share of items routed to review.

Common mistakes

• Review without feedback. If corrections are not captured and analyzed, the system never improves and reviewers burn out.
• Thresholds set once and forgotten. Revisit them as data accumulates, in both directions.
• Trusting the model's self-reported confidence alone. Combine it with verifiable checks.
• Too much review. If every trivial item needs approval, people stop reading. Save review for things that matter.
• Unclear ownership. Someone needs to own the queue, the thresholds and the decision to expand autonomy.

A quick design checklist

1. List every step in the workflow and mark which ones involve AI judgment
2. For each AI step, rate cost of error and reversibility
3. Assign a pattern: draft and approve, confidence routing, approval gate, limits, or sample review
4. Define the checks that decide when an item needs review
5. Design the review screen and where it lives
6. Decide what gets logged and who can see it
7. Agree on the evidence needed before a step gets more autonomy

Building it with us

Every AI workflow we build includes these review points by design, sized to the risk of each step. If you are planning an automation and want to get the control points right from the start, read how we work or tell us about the process you have in mind.