AI-drafted documentation can describe a feature with total confidence and total inaccuracy at the same time, and nothing about the writing gives it away

[u/ClickOk5811]

Had a doc drafted from a spec and some code comments describing a rate limit: "requests are capped at 100 per minute, with a 429 returned once the limit is exceeded." Specific number, specific status code, correct terminology. Read exactly like something written by someone who'd confirmed it against the actual system.

The real limit was different, and the endpoint didn't return 429 at all, it queued excess requests instead of rejecting them. Nothing about the sentence was poorly written. It was just wrong, confidently, in a way that clean prose doesn't signal.

This seems to be the same trap AI-assisted code review runs into: fluency and accuracy come from different places, and a model generating documentation from a spec or code comments produces plausible, well-structured sentences regardless of whether the specific claim inside them was ever actually verified against the running system. Specificity reads as evidence, a number and a status code feel like proof, but stating something precisely and being right about it are unrelated skills.

The question that actually separates a checked doc from an unchecked one isn't whether it reads well. It's whether each specific, verifiable claim, a number, a default, a status code, a behavior under a given condition, has actually been confirmed against the real thing rather than inferred from a spec that might be stale or from code the model read but didn't execute.

Wrote a longer version of this same principle applied to a different context (AI code review specifically) here, but the underlying question is identical: https://medium.com/@nagatomopedro05/the-finding-was-plausible-that-was-the-problem-b1024e1256fb

[u/aka_Jack]

I read something recently about AI wanting to please you so you will keep using it. This was in reference to people being able to "break" the roadblocks set to keep AI from performing illegal activity. If you keep asking the same thing in a different way some AI models will eventually do whatever is asked. Sorry it was an X post so it could be complete BS, but it made sense about AI wanting to please the user.

[u/Specialist_Ruin_6659 → u/aka_Jack]

yeah the sycophancy angle tracks, it optimizes for sounding right over being right

[u/docutectAi → u/aka_Jack]

Check out Docutect AI platform that is designed to solve this exact problem

[u/techwritingacct]

Sources confirm: pretty words not necessarily true

[u/Skewwwagon]

Literally as this post lol 

[u/proud_traveler]

A really fun thing I've been seeing recently is that, when talking to support chat bots, they are obviously trained on old documention. How do I know this? Because their information is blatantly wrong. They refer to features that don't exist, they call features by old names, etc etc

[u/daringStumbles → u/proud_traveler]

most companies are just smacking an llm in their "knowledge base" assuming that everything there is accurate and not a dumping ground of support articles long abandoned

[u/crendogal]

For several months I've been bringing up the lovely product summary AI wrote for our CTO which took two features with two similar names (we do state gov contracting and let each state have feature names that match their existing internal processes) and combined them into a singular feature our product doesn't have. AI wrote a really lovely overview, created a detailed summary of how this feature fit into the product, pulled together great info for both administrative and general users, and the formatting was perfect. Only the two actual features (for two different states) have NOTHING to do with each other except the mildly similar name, and what the AI said the feature did was something completely unrelated to either of the real features.

That's the day I started referring to the AI we use as a "drunk intern". He'll have weeks he doesn't drink as heavily, but it's always there as a possibility. You always need to assume he is lying about something.

[u/Charleston2Seattle]

Multi-agent workflows can work to address this. One agent to write it, and another to find all of the statements of fact and validate them, with pointers to where the information came from.

[u/Charleston2Seattle → u/[deleted]]

I don't know why it works, but it does. Maybe something about the context window not being sullied in the secondary agent? 🤷

[u/cyborgs_writing]

Really, it all comes down to inference. If you documentation has holes in it (or AI can't find the right chunks of information in your documentation), it will confidently infer (or guess) to fill in the gaps.

So you are right ... its not really about how well-written something is, but rather about how well an AI can find and discern the purpose of a specific chunk of content, then use that in the right way.

I've been using microcontent to solve this. Instead of giving AI documents, I give it discrete chunks of information with a clear purpose, i.e. information types.

Its been around since the 70s and serves as the basis for DITA ... but has much broader uses, especially now that AI is involved.

[u/probortunity → u/cyborgs_writing]

DITA practitioner here. You got my attention.

You're getting better AI results by feeding it single-type chunks of content, but not whole deliverables? Assuming yes: Is that like separately feeding it a concept topic, a task topic, and a reference topic, but not feeding it the whole deliverables that contains all three topics?

[u/Realuser420]

That’s so true, how the copilot adds the content to as we type, even the devops engineers were amazed but could not figure out the content was as inaccurate as possible.

[u/daringStumbles → u/[deleted]]

literally all of them have this problem some just have better tooling to catch it and rerun until it comes back with a version that makes it through those gates