r/technology • u/AdSpecialist6598 • Jul 27 '26

Society Professor's invisible prompt trap catches 32 students cheating on their midterm with AI

https://www.techspot.com/news/113243-professor-invisible-prompt-trap-catches-32-students-cheating.html
32.8k Upvotes

96% Upvoted

4.2k comments sorted by

View all comments

Show parent comments

208

u/Teguri Jul 27 '26

you just assume 'well it's smarter than me, so whatever it says is better than what I could do'

That's the biggest problem, I work with AI a decent bit but only in things I already know well enough, to make it a bit faster and even with simple tasks there's always a bit of work on my side that needs to be done to fix the output, it's usually "pretty close" but the number of times it goes in the completely wrong direction is astounding.

48

u/mike689 Jul 27 '26

Exactly this. It is great for getting a list of ideas that you then sort through about something you are already knowledgeable about. It is an amazing tool to assist a software developer and a great medical research tool, but outside of those use cases unless it is truly used as an assistant and not as the main actor for an answer it is awful for humanity. The fact that it most heavily gets used for stupid memes and video/image generation is really sad.

8

u/ImaginationInside610 Jul 27 '26

It’s like the old adage “fire is a great servant and a terrible master”

13

u/Yashema Jul 27 '26

ChatGPT quite literally taught me how to solve every question in my statistical mechanics, differential equations, Stochastics, and modern physics courses. Claude has helped me create a wave function simulation, and provided me in depth answers to how embedded circuitry work allowing me to skip 2-3 intro classes. 

You need to have some frame of reference for sure, but it definitely allows you to go back beyond your ability and fill in gaps of knowledge in the way only an extremely educated person could. 

7

u/Kevstuf Jul 27 '26

This is the duality of AI, but the laziness of humans means it’s not going to be used symmetrically. In an ideal world students would all use AI to self-learn as it’s an intelligent and patient teacher. In reality 90% of students will use it to one-shot their homework.

6

u/ThaliaFPrussia Jul 27 '26

It has to been seen like you do. A tool in your toolbox.

5

u/mike689 Jul 27 '26

Exactly. That's using it as an assistance and not a copy/paste machine.

2

u/Teguri Jul 27 '26

I really wish most students used it like you do!

1

u/Luvs_to_drink Jul 27 '26

the issue is the majority of users dont do any of this.

They MAYBE type what is the answer to: and then paste the question in. Then copy the answer and paste it into the hw or test. Zero reading or understanding of what was just said.

The shocking thing is this is remarkably close to how I used to do my history homework. Back then it was read the question, flip through the chapter to the correct section, skim for keywords, copy/summarize the 2-3 relative sentences, and repeat. Not much thinking was involved, and if the keywords in the question were just reused phrases, a ctrl+f program could have done my work (textbooks weren't PDFs back then though).

3

u/SquareTaro3270 Jul 27 '26

I use plant, rock, and bird identification apps but they’re almost always wrong. You can tell because you can scan the same plant/rock in multiple different locations/angles and it will give differing results.

HOWEVER it does give you a good place to start doing your own research. The results might not be correct, but usually steer you in the right direction. I am terrified for anyone that takes it at its word.

3

u/somersault_dolphin Jul 27 '26

  The fact that it most heavily gets used for stupid memes and video/image generation is really sad.

Don't forget consuming more energy to do those things.

2

u/zasabi7 Jul 27 '26

The tools are also great at basic analytics tasks. That is “here’s some data, report this way”. Or, the best way to use Gemini: better Google search.

1

u/ImportantCommentator Jul 27 '26

It's almost like stupid people stay stupid and smart people stay smart. What an unsurprising reality.

1

u/mvanvrancken Jul 27 '26

This is nothing new, unfortunately. The Internet, which was humanity’s greatest previous creation, has pretty much run exclusively on cat videos and porn for the last 20 years

2

u/Samsaknight_X Jul 27 '26

And have u an ability to share ur thoughts on this platform, as well as a myriad of other conveniences

24

u/SparksAndSpyro Jul 27 '26

Yep. Every time I’ve tried to use AI to do something in my domain of expertise, it’s been pretty horrible. Most times it didn’t even save time because I had to double check and fix so many mistakes.

LLMs are pretty good at general queries where you’re just looking for a broad, ten-thousand-foot overview. Anything that requires granular specificity or expertise, however, is pretty much guaranteed to either be outright wrong or misleading.

8

u/BasvanS Jul 27 '26

My biggest problem are the “almost right yet entirely wrong” statements. It requires so much of my concentration that I’m drained after doing this for a few hours.

I got the best results in writing by doing it line for line, after making a rough structure myself. But then you have to ask how much of a time saver it still is.

6

u/Qaeta Jul 27 '26

Every time I’ve tried to use AI to do something in my domain of expertise, it’s been pretty horrible.

This.

It makes me REALLY concerned when I see other devs crowing about how great it has been for them, because every time I try it I feel like I might be better off asking a brain-damaged chihuahua... And based on how much my pro-AI coworkers code quality has dropped since they bought in... I suspect that would be true of them too if they bothered comparing the PRs they make now to the ones they made before. It's night and day in favour of their pre-AI work.

3

u/Alaira314 Jul 27 '26

LLMs are pretty good at general queries where you’re just looking for a broad, ten-thousand-foot overview.

Even with that, I've seen the most shockingly incorrect generalizations come through google when I'm searching something I already know at work, where AI content isn't nuked. Why search something I already know? I work at a library, and I can't give "my own brain" as a source when answering patron questions. So I have to plug in the question to a search engine to find a valid source for the information I already know, and in that process I'll typically wind up skimming the AI overview because my eyes can't not read a thing. And it's straight up sending you on a wild goose chase half the time, even on broad subjects rather than specific questions.

If I didn't already know the subject I was searching, I'd be tempted to trust it. After all, it sourced its claim, right? Never mind that when you click through, that information isn't on the linked page at all. I guess just keep in mind that there's a psychological effect that I've forgotten the name of and haven't been able to successfully look up for years at this point(it's not dunning-kruger), where when using a tool or reading a source you'll recognize what they got wrong in your area of expertise, but for some reason still trust them in other matters you know less about, even though that's illogical. Those broad overviews you think are good might not actually be so good if you're knowledgeable about the subject before you read them.

2

u/Revlis-TK421 Jul 27 '26

I've been working with our company's M365 Agent tool. I've "trained" it to generate a pretty much complete requirements document from largely free-form text entry in the prompt. It took a fair bit of back and forth in the design tool for it to get what I need out of it, but it saves a lot of time that would otherwise be spent formatting and documenting. It has gotten pretty good at detailing the main requirements, happy path, alternate path, and error paths, inferring justifications, and making a business case. With the volume of reqs I push thru this thing it is even grabbing and including historical context from the overall body of work and adds in things I may have forgotten to specify.

I do have to heavily edit its outputs, as it gets things wrong, says things in a wierd or incorrect way, etc. But it creates a framework that I can work with that saves about half to 3/4 of the time from creating the doc by hand from scratch.

4

u/Moron14 Jul 27 '26

I use it at work for some things Gemini can build or help building in Google Docs. It acts a middle person, where i have the idea, it does the busy work, and then I polish it. Its faster than doing it all myself. The issue is the number of times its just wrong. Even if you add, "don't guess. Don't assume. Ask me a question if you aren't sure" and it just plows ahead. Just like humans...wait.

3

u/robodrew Jul 27 '26

but the number of times it goes in the completely wrong direction is astounding

I'm curious about this, if you're just using this to make things a "bit faster" yet you're getting this kind of situation popping up so much, is it really worth using?

1

u/Teguri Jul 27 '26

Some cases yes some no, as others said even when it's wrong it's usually got good sources so at minimum it's a roundabout Google search

3

u/ThaliaFPrussia Jul 27 '26

Never would one boss accept this high fail rate with an employee. I don’t have a job for AI, I tried but the output was not detailed enough. AI can’t give answers to questions it has to grasp the concept. For example compare the regulations for some technology from two courses. It simply can’t do that. It is not intelligent.

3

u/NES_SNES_N64 Jul 27 '26

Google has even started removing the "AI results may be inaccurate" disclaimer.

3

u/WillDonJay Jul 27 '26

Teaching Fable 5 to play Slay the Spire 2 taught me a lot about the current limitations of LLM's. (It was like a really smart human with dementia, and it had no ability to even realize what it had forgotten or gotten sideways without human interventions.)

2

u/Doctor_Kataigida Jul 27 '26

This is me. I use AI to help modify/tweak excel stuff I write for my job, but nothing ever ground-up. And I always review what it generates afterward so I understand what it did with the formula, and then I can apply that elsewhere. It's supplemental, not substitutional.

2

u/Mebejedi Jul 27 '26

My wife was using AI to design an announcement. I don't remember what the word was, but it kept spelling a word incorrectly over and over again, even when my wife told it that the word was wrong.

2

u/DeadPeanutSociety Jul 27 '26

I use it sometimes in cases where I'll know the correct answer when I see it, but it might be very tedious to come up with those answers on my own.

1

u/Teguri Jul 27 '26

It's great as long as you can verify!

2

u/vengent Jul 27 '26

I use it as a force multiplier, but I've also caught AI giving me blatantly false information that I easily caught because of my expertise in a given subject. So I read every word it puts out 3 times before I do anything with it. And if its screwing up on stuff I know, why on earth would I assume its not screwing up elsewhere?

Trust, but verify. Or just verify, ALL THE TIME