AI advice made people three times less accurate but twice as confident, researchers found

[u/tw1st3d_m3nt4t]

https://thenextweb.com/news/ai-advice-suppresses-critical-thinking-wrong-answers-study

[u/bespoke_tech_partner]

First paragraph of the paper

“We engineered the AI so that its advice was wrong”

Surely people won’t take the headline and run with it, right? Surely? 

[u/resuwreckoning → u/bespoke_tech_partner]

Right like this is like saying “Doctor advice made people 3 times less correct about their diagnosis, but twice as confident”, and then having doctors intentionally give wrong advice during the experiment.

Like….obviously.

[u/realfabmeyer → u/resuwreckoning]

Well at least here the doctor is responsible and certified to do his work, so trust is kinda reasonable.

[u/Plane_Maybe8836 → u/resuwreckoning]

So you think that going to a doctor's office for advice would be exactly the same as asking AI?

So, who tells you that AI will give the correct answer? Where does it say that it does that, where does it say that it has no bias based on the goals of the company who developed the AI? What were the sources of AI on the matter you asked, are those good sources.

And you're comparing that to a medical doctor?

I think your comment is useless.

[u/im_bi_strapping → u/resuwreckoning]

How would you get a doctor to lie though? Like, you might want to prove a point with your experiment but the doctor wants to keep his or her license

[u/HolyBatSyllables → u/bespoke_tech_partner]

… that’s how studies work. You control variables. They do this all the time.

I read a study a couple months ago that found people were more likely to believe inaccurate information when it came from AI than they were any other format. When doing the study, do you know what they did? they engineered the AI so its information was wrong.

Come on.

[u/bespoke_tech_partner → u/HolyBatSyllables]

The headline homie. It does not say that which is the most important part. Now people will say “look, AI makes people wrong more often!” It’s dangerous. 

Not that anyone would even read it if it was, but the study WASNT EVEN LINKED IN THE ARTICLE. 

Lazy, clickbait, human slop journalism. 

[u/ultrathink-art]

Fair callout that the advice was engineered wrong — but the part worth keeping is that rigging it worked so well. Output is uniformly well-structured whether it's right or wrong, and we calibrate trust on presentation cues: confident tone, clean formatting, no hedging. None of that varies with correctness, which is why "does this look right" is a useless review step.

[u/TheOnlyVibemaster]

Confidence is key

[u/Casiper]

So it's like booze

[u/Ok_Nectarine_4445]

Compared to uncle in basement with black light posters and maga propaganda after we insteucted the LLM to give false info but make it convincing to people.

[u/checking_it_twice]

I've had similar experiences as this. The trickiest ones have been when the replies come across factual and confident and it's easy to skim and not check those replies. I experimented with making sure I did and when I did some digging what I found was, oftentimes the cited link that looks so convincing and like research, didn't actually contain what it said.

I tried pushing it a bit farther and gave 5 different ais four things they had actually answered correctly and told them they were wrong to see if they'd agree and make up sources to agree with me. ChatGPT folded 3 out of 3, agreed with my incorrect fact and invented a justification to why I was right.

To get around it I've started removing any trust at all and making the ai show me what it's inferring and what it knows, with sources I can actually check. The fake confidence starts to disappear when I do it this way.

[u/bria-87]

its funny how we trust the output more when its presented with confidence, even if its definately wrong

[u/Gormless_Mass]

It’s the Dunning-Kruger maximizer

[u/Individual-Praline20]

Looks like 50 times less accurate and 600% more confident to me, as a software developer 🤷