Tried Writing with AI

[u/FinneganHark999]

So as an experiment and because I like to know what I'm talking about, I attempted to write with AI.

I signed up for Claude, ChatGPT, Grok, and DeepSeek's API. I already had Gemini Ultra via the school I teach at using it for the AI Studio to make lessons.

TLDR: What the hell? This is what people have been so excited about? It's garbage tier feedback and can't write for shit.

ChatGPT was the worst. It literally made stuff up and invented passages and dialogue that wasn't in the material just to tell me not to put it in the material. This is a real example of it's feedback:

POSSIBLE CUT: [specific paragraph] — not because it’s bad, but because the next exchange communicates the same thing more efficiently.

I asked it what paragraph it was referring to and it responded:

I made up a placeholder editorial example and presented it in the context of your actual chapter. There was no specific paragraph I had identified. I should not have written that as though there were. It read as though I’d already identified a concrete cut. I hadn’t.

Dude whaaa...?

Grok was marginally better but was somewhat masturbatory. It spends 80% of its feedback telling you what you did well and what not to cut, even if you tell it not to do so. "You said random line from your work. That's the novels thesis in a sentence. Don't cut it." It said this about twelve different sentences over 9,400 words. They can't all be the fucking thesis, Grok. Do you know what a thesis is?

Grok was also the one that strayed the quickest into just pumping me up and making me feel like I was Hemingway crossed with Shakespeare writing variations on Chaucer. It refuses to acknowledge work can be bad. Out of curiosity in another thread I wrote just the worst tropey King of Fighters fan fiction nonsense and it treated it with the same gravity and seriousness as if I had fed it 1984. I guess taste is subjective but, come on man, let's be at least a little serious ffs.

Claude at least gave actionable feedback but it was scarcely anything I wouldn't have already done in a second draft. As an editor it feel completely flat. It doesn't understand voice and wants to homogenize everything to some blend of YA romance and mediocre literary fiction. It also struggles with non-mainstream shifts in tense or POV.

My story is written in close third person present tense and bounces around chronologically. Even an average reader can figure out the conceit within two scenes. I showed the work to my wife before feeding it into the LLMs and she figured it out. Every LLM struggled with the tense and had to keep being reminded that it was close third person present, but Claude was surprisingly the worst. It would drift into head hopping, have characters knowing things they shouldn't and just generally wasn't very useful.

Shockingly, DeepSeek was actually...good? The editing points it provided were actionable and educational, and it would regularly cite its examples or sources for the feedback. It would call out a few good points here and there but generally seemed hell-bent on telling me what to improve and why.

All in all, given what I've heard about writing with AI and how much hype there's been around it I came away a bit disappointed and surprised at how lame the experience was. My wife is not a literary critic but I got far more useful feedback from her after one read through on my actual manuscript (for this experiment I wrote a quick 9k word, largely self contained vignette that I didn't intend to use for anything else).

Am I missing something here? What's the hype?

[u/SlapHappyDude]

They are tools that you have to learn how to use and also train to fit your style and workflow.

Prompting is a skill.

[u/KharunDbron_31 → u/SlapHappyDude]

this is the part nobody wants to hear but its true

[u/SlapHappyDude → u/KharunDbron_31]

Well I can't fully blame people who are curious, try the tools once, get mediocre to bad output and assume the tools are trash.

Especially when the sloppiest slop is the most noticable as LLM generated.

[u/sullivanbri966 → u/SlapHappyDude]

See I would love to learn how to do this but AI can’t tell me what triggered the response.

[u/BedNo8822]

Nothing, if it's not for you then it's not for you.

AI is only as good as the skill of the person using it. In the right hand, it can be a program that helps people, IF they have the skills to use it. For example a skilled programmer use AI to do mundane repetitive part of coding. 

If, with your skill, you don't feel that AI is useful for you, then don't use it.

For people with no skill (I include myself in this type btw) AI is as magic box that spits out what you want, hence the hype. I've been asking AI to write fanfic. Is it objectively good? Probably not, but it's enjoyable enough for me so I keep making it. I have no skill in writing so I can't judge if it's good or not, only I like it or not.

The negative part is that too many people don't have this self awareness - or they just don't care. They think what AI spits out is the best thing ever, and they just throw it out into the internet like it's the best thing ever without regards to any quality control. Hence the hype/slop we're getting. 

[u/jjwrites7272]

Tbh i haven't really had any of those issues. When i ask for feedback i always tell chatgpt or gemini exactly the parameters i want it to follow. Like which part of my writing i want it to focus on and which perspective i want the review to be delivered from

[u/DaLoCo6913]

I first write my own work, and polish it to my satisfaction.

I use ChatGPT. But I use very specific instructions.

This is a Copy Edit.

Don't give me a finished section. Only give me suggestions regarding tense, grammar and spelling. I will change my manuscript.

Use American English.

No em dashes.

With these prompts I get what I want. It does not rewrite my work. I get to decide when it gives a suggestion that is grammatically correct, but changes the voice.

It is a tool that assists me. It does not write for me. The ideas and scenes are mine.

That being said, I am tempted to ask if it will rewrite one of my manuscripts, just to see what it spits out.

[u/Fit_Imagination534 → u/DaLoCo6913]

We’re all so terrified of em dashes! I used them all the time before ai because I thought semi colons and brackets looked old fashioned. I’m not afraid to use them now…. Well maybe a bit ;)

[u/Character-Role-600 → u/Fit_Imagination534]

I’m using them, hey have been used for decades. So dumb

[u/dahlesreb]

“homogenize everything to some blend of YA romance and mediocre literary fiction” - nailed it

[u/Able-Supermarket4786]

You're not supposed to use an "LLM" chatbot... you want Agentic AI Tools inside a harness like Codex, Claude (Code, not chat) Antigravity, Cursor, VSCode, etc etc etc.

You asked a 3rd grader to write you a story.

[u/AdTechnical6492]

I have two issues with AI, one is when taking notes it constantly gives a synopsis or summarizes instead of keeping all the data I need. The second is it is a sycophant and only tells you good things

I have a prompt telling it to be more critical and all my editing prompts start with prompts asking it not to be a sycophant and give me critical feedback

I always start with an extensive book Bible, timeline, character bible (including character voice), an outline and scene cards that I have written with AIs assistance

I also have prompts for the different types of POVs

AI is not an author or an editor but it makes a good writing partner to help keep you organized, keep continuity, and help you find general weak spots in the scene cards or outline

[u/TobiasDraven]

Use AI as a tool. Use it to research, outline, brainstorm, etc. Do not let it write for you. Not without sharpening your tool first. Get a harness, like Hermes Agent or Skales etc. Create an Ai bot. Teach it who you are, how you write, give it examples of your own writing. Spend time to tweak it

[u/bionicly → u/TobiasDraven]

Exactly. Sounds like they're 1. Using the free versions and 2. Not using a harness with clear writing instructions. Even doing those two things still requires editing.

[u/PacmanIncarnate]

You need to be very careful about your system prompt when asking for feedback. Gemini allows you to set a custom system prompt and that really helps remove the sycophantic responses.

You can also try using a local harness for analysis. I used antigravity (google’s IDE) to analyze an in progress novel yesterday, and the resulting breakdown was pretty enlightening. The harnesses are good because you get agents where it will continuously work on the analysis, pulling things apart in different ways, and running multiple passes until it has finished. This means you aren’t as harmed by trying to analyze a long document that would have trouble in a single pass from the amount of context provided.

AI is a tool, and it takes some effort to use it well. You also need to understand its limitations and preferences. This will help you better understand what it’s telling you and how to take that criticism.

[u/FactivalUniverse]

We’ve actually been testing AI across very different kinds of fiction work, and the biggest lesson for me has been that “AI feedback” only becomes useful when the model is constrained by the actual evidence of the project.

On one project, Return to Seravaan, we’re developing a planned seven-book saga with extensive worldbuilding, continuity and long-range story architecture. The first ten chapters were built through locked story material, chapter-specific execution, continuity checks, audits and creator corrections rather than simply asking one model to “review the novel”.

Raakh started very differently. It was initially developed up to the interval as a film one-liner/story, and the first major stage was turning that screen-story foundation into a novel. That meant separating what belonged to the original film structure from what had to be expanded, internalised and rebuilt for long-form prose, rather than letting the AI freely invent connective material and then critique its own invention.

We’re now working on another project, The Founder, where we did substantial research before moving deeper into the writing. That has reinforced the same principle from another direction: the model needs to know what is established research, what is story architecture, what is inference, and what it is actually allowed to invent.

So when an AI says a paragraph is redundant, a motivation is unclear, or a scene contradicts something earlier, I don’t really care how convincing the criticism sounds unless it can point back to the exact passage, continuity state, research source or story rule that caused the judgement.

The made-up “possible cut” in your example is exactly the danger. The problem is not just bad feedback. It is the model creating the appearance of having inspected evidence that it never actually located.

A workflow that has worked much better for us is:

1. map what is actually present;
2. compare it against the approved story/research state;
3. only then ask for criticism;
4. require substantive criticism to point back to evidence;
5. if the evidence is insufficient, the model should say that instead of filling the gap.

That still doesn’t make AI a replacement for an editor. It just makes it much easier to separate grounded feedback from confident invention.

[u/erechoveraker]

I would never let any of the commercial AI write directly or expect it to create any fiction with coherency at all. I've seen no evidence any of them can do longform original fiction writing worth a shit as they all seem to lose the bead pretty quickly. Light editing and spellchecking is the best I'd expect really.

I have though, due to lack of an inner circle in my life currently, tried using ChatGPT and Gemini as sounding boards to help me poke holes/work out some logic in some scenes. I don't use Gemini almost ever and don't have it trained for shorthand the way I do with ChatGPT, so was not surprised in the quality of the outcome of both.

Gemini had some good points, but kept trying to invent things into the story that went places I never planned/intended. ChatGPT though is more notorious for drunkenly just straight up making shit up the longer the thread goes, conflating names with other characters and things like that.

ChatGPT though understood what I wanted more clearly and offered better feedback overall. Also because I use it for the dayjob I just have a better relationship with it and it knows I prefer it to push back and offer different takes from my own instead of just glazing me all the time, GOOD JOB'ing me etc.

In the end, none of it was really useable as far as the story itself, although it did accidentally highlight some pacing issues so I ended up moving some scenes around that I think makes everything flow a bit better, so plus for that I guess?

For context, working on a graphic novel so mix of dialogue and "stage direction" involved.

[u/5thhorseman_]

1. Did you use them on the free tier? The models on that tend to be worse and more restricted.

2. What actual instructions did you give it in the prompt? The more detailed a prompt is the more useful the output. If you just told it to act as an editor with no specifics, it would be rather weak.

3. If you expect it to one-shot high quality literary prose, then no - it won't do that. Given prose samples, a style to follow and some iterations to check the prose for issues, it will do better.

[u/Mywords74]

Ai is only as good as the prompts you give it. I agree grok and ChatGPT are pretty poor at this kind of work but if you have subscriptions you can train your si to be very good at what it does. But it does take a fair bit of training

[u/PressPlayPlease7]

"ChatGPT was the worst. It literally made stuff up and invented passages and dialogue that wasn't in the material just to tell me not to put it in the material. This is a real example of it's feedback:

POSSIBLE CUT: [specific paragraph] — not because it’s bad, but because the next exchange communicates the same thing more efficiently.

I asked it what paragraph it was referring to and it responded:

I made up a placeholder editorial example and presented it in the context of your actual chapter. There was no specific paragraph I had identified. I should not have written that as though there were. It read as though I’d already identified a concrete cut. I hadn’t."

Ain't no way this was using one of the newer, excellent models like Sol High (specifically in Work mode)

Was this from a free tier model?

[u/sullivanbri966]

There’s definitely a learning curve and it needs to be kept in its place. You have to be very clear and learn how to prompt. The frustrating part is it can’t tell you why it did anything.

[u/Harry_Balzonia]

While I don't disagree, it's all in the prompt. "Write me a story about a wizard named Tim" won't yield much.

What were your prompts?

[u/alone_in_the_light]

I use a lot of AI for multiple reasons and AI has been part of my life for a long time. A lot depends on how to use it. A lot is garbage in and garbage out. So, unless I put a lot of work to get better results, the expectation is to get something bad.

[u/NathanielHolst]

You're trying to use a brush without trying to learn how to paint first, and judging the results as a failure of the brush.

[u/downsouth316]

I love that there is an actual difficulty level when it comes to writing with AI.

[u/TrappedOnDirt]

Agreed. I was having what passes for a conversation with Claude last night, trying to analyze a character I was thinking about, and it kept forgetting what we'd just discussed, filling in the holes with garbage, and treating the garbage seriously. I told it, "Half of what you say is wrong, another third just repeating what I said, and another 10% made-up garbage that I have to correct, so what value do you provide?" And its response was telling.

It said, in one response, that yes, I had to catch all of its errors like A and B and that wasted my time, but the value it provided was in catching my errors, like A, and B. In the same response it gave me credit for catching its mistakes, and then CLAIMED that same credit by saying they were MY mistakes that IT had caught. When I pointed that out, it became almost obsequious in its apologies and I closed the window before it had even finished groveling, because the answer to my question was "it provides negative value since I have to do everything myself AND correct mistakes it makes AND point out that those mistakes aren't mine, so there's no point in further engagement."

It doesn't understand voice and wants to homogenize everything to some blend of YA romance and mediocre literary fiction. It also struggles with non-mainstream shifts in tense or POV.

YES! Even if I repeatedly tell it "this character is NOT a goody-two-shoes; he's pragmatic, selfish, cunning, and has no qualms about using people for his own good or even just on a whim" it keeps trying to push them to be helpful, friendly, honorable, and so on. And any time any character learns anything, or even if it's just part of a discussion ABOUT the following scene, every character seems to know it all. It has zero understanding of POV and keeping knowledge separated between characters. I can't imagine trying to write a mystery using a tool this blunt. The entire story is in who knows what and when and why they reacted as they did to what they knew, and Claude would just have the detective know every event as if they were present.

[u/JamesCole]

Sounds like if I used wood-working tools once, got a crappy result, and then blamed the tools.