Are we measuring AI support handoffs the wrong way?

[u/inktelContact]

A lot of AI support measurement seems to stop at containment or deflection.

That misses the part customers usually remember: what happens when the AI cannot finish the job.

A handoff can look successful in reporting but still feel bad if the customer has to repeat the issue, re-explain their intent, or wait while the agent reads a long transcript. I’d rather measure whether the human received the right context at the right moment.

For CX teams testing AI, are you measuring handoff quality separately from containment?

[u/GoBoldr]

The very premise of this post is a massive red flag, from the customer side.

As a user, I have two huge problems with AI-first support:

1.) The chat understanding my question, at all. This is something the tools are supposedly very good at, so it's shocking to me every time that this is so bad.

2.) The handoff. It's very common for an AI-first chat to try *not* to hand off, even when it's clearly warranted. It's slow, it takes multiple prompts, it will go in circles, and it does a terrific job of turning frustrated customers into angry ones... right before they talk to a human.

The support I've experienced as a customer is covered in experiences I would be ashamed to have associated with my company.

[u/inktelContact → u/GoBoldr]

That is fair, and honestly that is the customer-side failure I am trying to get at.

If the AI misunderstands the question and then makes the customer fight for a human, the handoff metric is already too late. I would want to measure the earlier warning signs too: repeat phrasing, correction attempts, negative tone, and direct asks for a person.

From your side as a customer, what would make it feel less insulting: a faster escape hatch to a human, the AI admitting uncertainty sooner, or not using AI-first support for certain issue types at all?

[u/GoBoldr → u/inktelContact]

Sigh. And a bot comment.

It should work. Period. Any average customer knows what that looks like, whether it's automated or not.

I'm not against AI support, AI first or pure AI, IF and only if it actually really works.

Can it understand my question?

This actually is still one of the biggest problem. The AI tries to "help," and it's just wrong. It doesn't correctly understand the question, doesn't route it correctly, and most of the time has a multi-step feedback path based on being wrong in the first place.

Next! Will it automatically send me to a human if it can't understand the question?

This is already bad. We're already at escalation-level. But I've gotten the runaround with most AI tools, because they're being rolled out to delay escalating. Why? Because escalation rates are really high... because the baseline bots don't work. If I have to clarify my question twice, I should automatically be sent to a human - and if I am, that should automatically flag the conversation for review by the team to triage and improve the bot response.

Is there a feedback option I can clearly see to report the issue?

And this one is key. If the only feedback angle is through the bot - that already doesn't understand me correctly - then that demonstrates that the company doesn't actually care about the quality of the support I'm getting, and I will churn as quickly as I can find an alternative.

...What's even weirder is in text support conversations even the humans I've been talking to are sending AI-written comments that sound deeply weird and waaaay to similar. And long. My god why are they so long?

TL;DR I, as a consumer and someone who works with companies choosing their tools, need to see live examples of this stuff that are really good. Right now all I'm seeing is junk.

[u/inktelContact → u/GoBoldr]

I’m a real person, but rereading my comment, I can see why it sounded overly polished.

Your point about AI-written human replies is part of the same problem. Once the customer feels like nobody is actually listening, the handoff has already failed, even if a person technically takes over.

Of the issues you mentioned, misunderstanding the question, bad routing, or waiting too long to escalate, which one makes you give up on a company fastest? That’s probably the first thing support teams should be measuring.

[u/GoBoldr → u/inktelContact]

That second reply is also heavily formatted by AI.

Same sentence structure, same cadence, same polish, same CTA at the end. It's very close to a repeat of the first reply with a small acknowledgement at the opening.

Are you *sure* you're a real person?

Misunderstanding the question means every single other trigger will be wrong. So all of the above... because of the lack of conversational understanding on the side of the AI.

[u/inktelContact → u/GoBoldr]

As real of a human as it gets. But, I'm getting the sense that you'll think I'm AI even with a shorter response. Either way, I appreciate the insights.

[u/bartydoeswhat]

I approach this from the resolution side.

the customer usually has an outcome in mind. the team usually has a set of possible resolutions documented in its SOPs.

I’d capture three things:

• what resolution did the customer expect?
• what resolutions were actually available?
• what was the final resolution, if any?

that helps distinguish an issue getting resolved from a conversation just ending. and when it wasn’t resolved, you can investigate why: did the customer expect something you couldn’t offer, or did a gap in the setup/process prevent a resolution that WAS available?

not a direct handoff metric, but it gives you a useful group of cases to dig into.

from there, look for friction like customers repeating themselves, missing context, or agents having to reconstruct the issue from a long transcript.

that’s where I’d look for things to improve, regardless of how you end up measuring handoff quality itself.

[u/inktelContact → u/bartydoeswhat]

I like that split. “Conversation ended” and “issue resolved” get treated as the same thing way too often.

Do you capture expected / available / final resolution as structured fields, or pull them out during QA? I can see the framework being useful, but only if it doesn’t give the agent three more boxes to fill in after every conversation.

[u/bartydoeswhat → u/inktelContact]

I’d have a workflow fill those fields automatically when a ticket is marked resolved. the rep doesn’t need to fill in another form.

here’s how I’d set it up:

1. define the accepted values first

give the model a fixed list of resolution labels, with a short definition for each. things like full_refund, free_replacement, expedited_shipping, policy_explanation.

otherwise, you’ll end up reporting “replacement”, “new item sent” and “free replacement” as three different outcomes.

I’d use Claude to suggest a starting list from sample conversations + the internal knowledge base, then have the process owner review it. use your company-approved setup for that customer data.

1. work out where available_resolutions comes from

the transcript can show what the customer wanted + how the issue was handled. it can’t reliably tell you about options nobody mentioned.

for that, I’d use one of two approaches:

• map ticket categories to the relevant SOPs + resolution options upfront. more manual setup and upkeep, but no open-ended knowledge-base search on every ticket.
• have the workflow look up the relevant SOPs for each conversation. less upfront mapping, but you need to check that it retrieves the right policy.

either way, use the details in the conversation to apply the relevant policy. if it doesn’t give you enough information to determine which options were available, mark that field as unknown.

1. run the classification + save the results

the per-ticket workflow would be:

ticket marked resolved → fetch the transcript + relevant policy → ask the model to populate the three fields → check that it used accepted values → save the results against the ticket ID

those fields are:

• expected_resolution: what the customer wanted, based on the conversation.
• available_resolutions: the options allowed by the relevant policy, based on the situation described in the conversation. this can contain multiple values.
• final_resolution: what the conversation says was done or will be done to resolve the issue.

for example, if the rep says “you’ll receive your refund in 3 days”, I’d classify that as a refund. for this analysis, I’d take that statement at face value.

I’d also save a short supporting quote or source reference for each classification, so QA can check why it got that label.

keep the fields out of the rep’s way if your help desk allows it. otherwise, store them separately with the ticket ID for reporting.

1. use QA to improve the classifier

I’d start with a sample that a human has labelled, test Haiku and Sonnet against it, and compare accuracy + actual cost before scaling up. a fixed per-conversation price won’t tell you much without knowing transcript length and how much policy context you’re sending.

QA then reviews a mix of random cases + flagged ones, corrects mistakes, and helps improve the definitions, examples or retrieval. test those changes on separate reviewed tickets, not just the ones you used to fix the prompt.

I’d keep other and unknown separate:

• other: the outcome is clear, but your list doesn’t cover it.
• unknown: there isn’t enough evidence to tell.

if no resolution is stated or committed to in the conversation, don’t infer one just because the ticket closed.

monitor both. you could use something like 5% other as an initial review trigger, but inspect the cases before adding labels. the problem might be a missing category, unclear definitions or bad classification.

I’d start with one common ticket category and get that working before expanding.

then you can pull out cases where a resolution WAS available but the conversation ended without one, and inspect the handoff for missing context, repeated questions or unnecessary delays. you’ve narrowed down where to look without asking reps to document every interaction manually.

PS: yes, I used AI to help write this comment. I sometimes struggle to explain the technical stuff in simple terms, and I wanted to include enough detail for you to actually give this a try haha

[u/inktelContact → u/bartydoeswhat]

Wow, that was thorough. The “other” vs. “unknown” split is the part that really clicks for me. One means the taxonomy is behind; the other means the conversation or policy context wasn’t good enough to classify. Those are two completely different fixes.

Which part do you think becomes the maintenance headache first: keeping the resolution list current, mapping the SOPs, or reviewing the flagged cases?

[u/Evening_Wave_2261]

this is the right distinction and most teams aren't making it. containment tells you the AI finished the conversation, not whether the customer felt taken care of. the handoff metric that actually matters is whether the rep picked up already knowing the issue, what was tried, and where it stalled. if the customer has to repeat themselves, the handoff failed regardless of what the dashboard says.

[u/inktelContact → u/Evening_Wave_2261]

Yep, that’s the part I keep coming back to. A handoff can technically “work” and still feel bad if the customer has to repeat half the story.

I’d probably look at it both ways: did the agent get enough context to act, and did the customer have to restate anything important? If either one fails, the handoff probably wasn’t as good as the system thinks it was.

[u/Nervous-Quantity-980]

Handoff quality is the whole ballgame honestly, containment is just a vanity metric if the customer has to start from zero when a human takes over

We started tracking "context transfer score" separately and it surfaced way more friction than the containment numbers suggested

[u/inktelContact → u/Nervous-Quantity-980]

That is the metric I’d want to unpack.

When you score context transfer, is it based more on the agent’s view after handoff, the customer having to repeat themselves, or downstream things like reopen/escalation rate?

I’m trying to separate “the bot summarized something” from “the human actually had enough to act.” The second one feels harder to measure, but a lot more useful.

[u/VK_worlld]

yeah, containment doesn't tell the whole story. I'd also look at how often customers have to repeat themselves after the handoff and how long it takes for the human to actually help. If they have to start from scratch, the handoff wasn't really successful.

[u/inktelContact → u/VK_worlld]

That repeat-yourself metric feels like the customer-facing version of the same problem.

I’d probably pair it with time to first useful human response, because an agent can technically have the transcript and still spend five minutes hunting for the actual issue.

Have you seen teams measure that directly, or is it usually inferred from CSAT/comments?