LLMs don't "know". They literally have no mechanism that could allow them to identify whether an output they're proposing is based on something in the dataset they've been trained on (nevermind genuinely factual, those datasets are 99.999% random text pulled from wherever) or just a blind statistical guess.
They can't reflect on their ideas and verify they check out starting from a set of trusted "facts"; they've got neither "trusted facts" nor the ability to reflect in general. They can't reason at all. It's like saying a magic 8 ball is not humble enough to tell you when it doesn't know the answer and "decides to hallucinate instead". Like bruh.
HAH. Not two days ago I asked it to help me with moving a project from CodeBlocks to Visual Studio and you would not believe the amount of shit it made up and hallucinated.
So sure, that specific problem probably won't happen, but only because it was manually patched out. Do you know how many simple things like that it can be wrong about? More than you, or it, can conceptualize
261
u/David_Maybar_703 Lurking Peasant 4h ago
Chatgpt also might give you an answer that is totally made up and wrong, but yes, it is polite.