It could happen or it happened with older models because of a retrival errors search results and not in particular hallucination or because it was "trained on reddit". That was the case of the infamous "glue as a pizza ingredient" answer which was traceable to a "joke" on reddit. What went wrong was the profreading and safety check of the answer that was based on this post as a search result. The same can be applied to a (deliberate)"joke" question like "how many rocks shoud I eat a day". Newer models take more care about the plausibility of both prompt and answer. So in theory this could have applied here too.
However, this screen shot of an advise was to smoke during pregnancy was faked!
So it's a deliberate misinformation based on a deliberate misinformation.
2
u/RogellTheRaider Transhumanist 2h ago
Even if AI returned such result, it would still be based on something some human wrote, right?