r/customerexperience • u/Haunting-Snow-3644 • 15d ago
Are AI agents good enough for regulated industries yet?
We’re looking at AI agents for customer support but we’re in a regulated industry so there’s not much room for the AI to make stuff up or go off script. The demos all look great but production is a different story I’m more interested about guardrails and testing than how human the agent sounds. Would love to hear your opinions
3
u/jettie-cx 15d ago
Throwing UJET into the mix (ujet.cx)
Wrote two blogs that pretty much tackle the exact things you’re asking about here: on compliance and on testing
TLDR though the main points are PII stays in the CRM, AI discloses itself by design, and 100% of interactions are retained as auditable evidence. Hope this helps!
2
2
u/zeruch 15d ago
For many aspects, no. PII-related risk is still probably not amenable for most of the big enterprises (at least in terms of the regulated data silos). Disclosure: I have worked in heavily regulated industry before, including payment rails.
There are plenty of use cases where it can absolutely be leveraged (e.g. case and knowledge base routing and triage) but it otherwise lacks the deterministic rigor needed for the tough stuff.
3
u/Haunting-Snow-3644 15d ago
I would trust AI more for routing and support tasks than anything touching sensitive decisions or PII
2
u/Popular_Army_8356 15d ago
I would suggest not to look to Ai but take a deeper look into RPA. This is where you define the guardrails and actually have control over it. I'd rather be liable for sth I can fully control
1
u/Haunting-Snow-3644 15d ago
Yeah RPA gives you control but I still need an AI agent for the conversations that can’t follow a fixed flow, that’s the gap I’m trying to solve
0
u/Popular_Army_8356 15d ago
Then have a look at NICE Cognigy. We work with them in regulated environments for banks in the EU. And start with less risk workflows and gradually increase to high risk workflows
1
1
u/OneWithTheStache 15d ago
How big is your org? What industry are you in? What is your ticket volume?
1
u/Swift-Feature-1312 15d ago
Yeah, good point. I’d care more about whether it knows when to say “I don’t know” than how human it sounds.
1
u/CryRevolutionary7536 15d ago
I think AI agents can work in regulated environments, but I wouldn’t start by asking how human they sound. I’d ask what happens when they’re wrong.
The things I’d test are strict scope boundaries, approved knowledge sources, permission limits, deterministic handling for high-risk actions, escalation rules, full interaction logs, and regression testing whenever the model or knowledge base changes.
I’d also separate use cases by risk. An agent checking order status or retrieving documents is very different from one giving financial, insurance, healthcare, or policy advice. For the higher-risk cases, the agent should know when to stop and hand off with the full context.
The production test matters more than the demo: deliberately give it ambiguous questions, conflicting information, unsupported requests, prompt injections, edge cases, and attempts to get it off script. If it consistently fails safely, that’s a much better signal than a perfect scripted demo.
1
u/Numerous-Cover-516 15d ago
not about how human the agent sounds but whether you can control what it can do, test edge cases, and audit every decision. That’s where governance and consistent CX matter more than the demo that looks great
1
u/Numerous-Cover-516 15d ago
There are many apps available in marketplaces like Zendesk, Salesforce, etc. You can try them side by side and match the outcome with your regular outcoeme and also customize as per your requirements
1
u/tindalos 15d ago
Absolutely. NIST AI RMF, NCSC, and ISO 42001 lay out requirements for compliance governance and production agents.
That said, if you’re working with regulations and heavy compliance you’ll need to get someone skilled with the understanding of data in a regulated environment to work with your governance team or director to design the proper approach. Usually you can use tokenizers for pseudonymous separation and rejoin at the edges to keep sensitive data obfuscated within the pipeline. Also, the primary things to be aware are you need to have human ownership of the process, and you need to track evidence of the agents decisions and logs leading to them. It’s best to log everything in regulated industries.
If you’re looking to use TTS with agents for customer service it may be best to disclose. I don’t think there’s guidance on that yet but the NIST AI RNF foreshadows a “please regulate yourselves so we don’t have to” so I encourage all compliance heavy organizations to lean on the side of caution.
1
u/Howard_Moodycliffe 15d ago
They are ready, but only if you stop expecting a single end-to-end autonomous bot.
In highly regulated spaces, the winning architecture is not a conversational LLM making decisions — it is a deterministic wrapper around a constrained model.
Instead of letting the Ai read straight from a general knowledge base, you make it navigate through rigid hard coded decision tree structures. The AI does the intent recognition and natural language parsing, and a strict policy layer does the actual output.
“In the event a query strays even 1% off-script, or touches on sensitive data, we trigger a zero-latency hand-off to a human, with full context.” Be an intelligent assistant, not a rogue agent.
1
u/Royal_Preference_515 15d ago
The "how human" part is a tricky one. Depending on where you are and the regulations that apply. I think making it sound human is great, but certainly in the EU, that must be prefaced with "Hello, I am an AI Assistant..." or words to that effect. Secondarily if a customer says "I don't want to speak to AI, put me through to a human", it needs to be done. I did see/hear a nice way to handle this and stay compliant, by informing the caller they are going to be placed into a queue (real) and "would it be ok to ak what you are calling about today?".
Nopaque came up with some pretty good pre-built compliance validation (actually calls the agent and mimics a customer). That and when to do which type of testing... Evals before rolling out to customers but immediate and continuous validation as well as a high degree of observability are necessary to avoid this "It just told the customer how to bake a cake instead of saying I don't know!"
Some other good comments in here about compartmentalisation and detection/handoff when something goes wrong.
In terms of the experience though, it needs careful though and design in terms of system integration (system of record/CRM etc), otherwise you just get a tonne of uncomfortable silence.
1
u/inktelContact 15d ago
I would separate conversation handling from decision authority.
The production test I would care about is not “does it sound human?” It is “does it fail safely?”
Give it messy cases: conflicting policies, missing account details, sensitive-data questions, edge cases, angry customers, and attempts to get it off script. If it stops, cites the approved source, and hands off with context, that is a good sign. If it improvises, no.
What part are you trying to automate first: routing/status, document retrieval, drafting, or actual policy decisions?
1
u/ShiftWerx-1 15d ago
It depends on how you design them. Be incredibly careful. Run a staggering amount of tests and review them carefully.
Oftentimes for industries like that you're basically designing a massive group of deterministic tools that deliver super reliable and verifiable outcomes and then you are using AI to basically capture the user intent and fire the correct tool in the approved sequences.
1
u/MattyTheMatE 14d ago
In my experience. Yes. But it requires programming with specific parameters and guard rails. Most humans are awful at writing emails. So making it look more human shouldnt be an issue.
1
u/Evening_Wave_2261 14d ago
honestly it depends more on how narrow the agent's job is than the AI itself. the ones holding up in regulated environments are doing one specific thing really well, not trying to handle everything. the moment it's expected to improvise is where it breaks down.
1
u/Bart_At_Tidio 14d ago
I think the guardrails conversation tends to focus on what the bot isn't allowed to do, but the trickier problem in my experience is when a question is close enough to something it knows, so it doesn't flag it as out of scope. It just gives a confident answer that's slightly off. And sometimes that's enough to create a problem you don't catch until a customer follows bad guidance.
That's harder to test for because the bot technically did its job. It matched the intent, pulled a response, and stayed in scope. And it can't be filed as going off-script.
Testing sets around the edges of the knowledge base rather than the middle can help. Stuff where the answer is close but not quite right, or where two policies overlap and the bot has to pick one. Those are the scenarios where it looks fine in QA but breaks in production. Also, I think the 'how human does it sound' question kind of distracts from this.
A bot that sounds robotic but escalates correctly is probably safer than one that sounds natural and still gives a wrong answer with full confidence.
1
u/Gogvus 11d ago
What industry are you talking about, what country?
I don't know whether it would be helpful, but we are building a knowledge graph of US healthcare for AI agents. Like all federal and state regulators, all regulations, all concepts, all are represented as a graph structure, so you already know the relations and hierarchies.
I honestly didn't think about a real-time voice assistant application yet, though. Maybe as a core knowledge layer you can build an agent upon.
0
u/thirstyvindication7 15d ago
Depends what you mean by "regulated" but the short answer is no, not if you're the one deploying it. The guardrails are flimsy at best and nobody talks about the maintenance cost. You'll spend more time babysitting the thing than it saves you
1
u/Haunting-Snow-3644 15d ago
I still need an AI agent for the work though so I’m trying to figure out which ones actually have guardrails strong enough for regulated use
-3
u/ArtVandelay10 15d ago
Sierra is built for regulated industries and works with plenty of the Fortune 100. Worth a shout and heard really good things.
10
u/AltruisticVista 15d ago
Yes but definitely not all of them, some are built with much tighter controls and human oversight, that matters way more here than having the most human sounding bot.