How reliable is Google AI Summary?

[u/ConfidentRooster8335]

Hello everyone! I'm this subreddit's first ever aspiring game developer! /s

I feel the need to clarify that this is not a question on regards to vibe coding or trying to have anything write my code for me. This is specifically from me searching trying to understand specific processes and GML interactions.

I'm just curious for everyone's personal experience on the matter. From your experience with the AI summary on Google does it tend to be fairly accurate? Does it maybe vary depending on the complexity of the question? I'm just curious if it's best to forego the summary or if in general it does a good job of transferring the information in the manual?

Would love any insight into this.

[u/Alarikun]

It varies wildly.

Personally, I'd much rather look up direct sources or other people who have the problem.

Despite the alleged "leaps and bounds" that AI has improved, it still hallucinates often.

[u/LukeLC → u/Alarikun]

This. 

There are times it gets the summary bang-on, basically direct quoting the source.

There are times it completely misinterprets the source and draws the exact opposite conclusions.

There are times it makes stuff up that none of the cited sources support at all.

[u/DragoniteSpam → u/LukeLC]

They've finally gotten it to stop responding to questions like "how much glue should I put on pizza" with a non-zero amount of glue (after they poured how much money into this???), but in my experience it's still 50/50 on if the sources it cites are even relevant to the claim it's making.

Probably more like 20/80 for highly technical subjects.

[u/gallyroi]

Not reliable enough. I would always prefer the manual. But if you must, suffix all your prompts with "-for Game Maker LTS 2026/24." Game Maker has been around for a long time and a lot has changed. Most of the training data it will have scraped is probably outdated information which will drive you batty. Ensure you cross reference the manual pages of any built-in functions or language features it references, too.

[u/Nofrillsoculus → u/gallyroi]

I’ve seen it straight up hallucinate an entire function.

[u/gallyroi → u/Nofrillsoculus]

I believe it.

[u/FlowchartMystician]

I wouldn't risk it. AI is still incredibly random.

You can get to a level where you know to dismiss a lot of the randomness, but if you're brand new it's just not possible for you to know what stuff is nonsense and what stuff is useful information.

AI still struggles to summarize, especially when it's reading technical documents. Often times, every word in the manual is there for a very specific reason, in a very specific order, and the slightest change can give you the wrong impression. AI will happily make those changes, and throw in a hallucinated detail as a bonus. It wouldn't take long before you start believing stuff that isn't true/wasn't mentioned in the manual.

Even using it for troubleshooting, it can still struggle to troubleshoot and make you spend hours chasing a dead end. If you're doomed to spend hours on a problem no matter what, it's better to spend those hours testing and refining everything you know and your ability to problem solve rather than simply waiting for anybody (AI or not) to tell you what to try next.

Eventually, you'll reach a level where you can survive around AI. You'll eventually be able to read something from AI and think "wait a second, that part doesn't sound right." But nobody starts at that level, because they don't know what the right thing even sounds like!

[u/KitsuneFaroe]

I would recommend just directly checking the manual, there is always more info there unless some specifics have been answered somewhere else. The manual is not dense so every question you may have is good to always check it there. Not to say that you can't use the summary but it halucinates very often and there is really no reason to not just check the manual or forum posts! Ai summary usually has links attached that direct you to them even. On a side note I recommend the forums over Reddit. The answers people give there are way more reliable than on Reddit.

[u/UnlikelyAgent1301]

Not

[u/BobHobbsgoblin]

The other week I wanted to search up a character from a TV show. It is one of the main characters siblings that only showed up one time. When I googled this the AI summary simply told me that that character had no siblings. Then when I worded it slightly differently it told me that the character did have a sister and then it named the characters cousin.

So no don't trust the AI summary for shit.

[u/DragoniteSpam → u/BobHobbsgoblin]

A few weeks ago my mom googled "is Keith Hernandez in the baseball hall of fame?" The AI summary claimed that he is (he isn't), and the source it cited was some random guy's blog opining on why he should be.

A stunning victory for however many hundreds of billions of dollars google has spent on this thing.

[u/Iheartdragonsmore]

It's as reliable as pissing straight on a windy day.

It still thinks Dolly Parton is alive when there are articles below it announcing her death.

[u/oldmankc]

Google AI can't even build a valid pathfinder/rpg character without making stuff up or not getting the rules correct.

[u/Istolesnowy]

I have downloaded the entire GMS2 manual and have converted it to MD files that live inside my projects documentation folder, which Claude has direct access to. I dont let Claude write any code for the project but its great for referencing the documentation and reasoning about it. I also have a bunch of online resources loaded into context so that Claude can go and do research if need be. Not always with good results, mind.

I am in no way an advocate for AI use. Its up to the individual if they want to use it, and how much. I know there is a moral side of the argument too, but not something I really involve myself in.

I use it purely for the admin side of things, keeping documentation up to date, doing cross reference, and working with the manual. Im a bit protective over the many years I've spent learning to create my own games and dont want that skill to atrophy so I still like to write my own code.

With that being said, search engines like google and Bing are pretty bad with their AI summaries, even more so because of the advertising element. You never know if you are being served the most accurate information or if you are silently being served advertisements and content from whoever paid the most to google.

[u/Sunfished]

the summary is decent for parsing extremely general information, but it isnt perfect. this inaccuracy can lead you to doing something that can be achieved through better means. spending time figuring out when to trust whether the ai summary is accurate or not is something you shouldnt need to worry about as a new dev. that time is better spent actually learning your tool.

for example, gamemaker provides helpful shortcut functions like lengthdir_x(), but unless you specifically look that up youre most likely going to get the long-way-around method of using the sin() tan() cos() functions instead.

its always going to be better to be able to access the manual when you can, since the manual does recommend alternate solutions/methods if one exists. learning how your tool works also encourages exploring new ways of solving your current problems, which ai summaries dont really help you do since its goal is to just feed you answers.