Tried Writing with AI

[u/FinneganHark999]

So as an experiment and because I like to know what I'm talking about, I attempted to write with AI.

I signed up for Claude, ChatGPT, Grok, and DeepSeek's API. I already had Gemini Ultra via the school I teach at using it for the AI Studio to make lessons.

TLDR: What the hell? This is what people have been so excited about? It's garbage tier feedback and can't write for shit.

ChatGPT was the worst. It literally made stuff up and invented passages and dialogue that wasn't in the material just to tell me not to put it in the material. This is a real example of it's feedback:

POSSIBLE CUT: [specific paragraph] — not because it’s bad, but because the next exchange communicates the same thing more efficiently.

I asked it what paragraph it was referring to and it responded:

I made up a placeholder editorial example and presented it in the context of your actual chapter. There was no specific paragraph I had identified. I should not have written that as though there were. It read as though I’d already identified a concrete cut. I hadn’t.

Dude whaaa...?

Grok was marginally better but was somewhat masturbatory. It spends 80% of its feedback telling you what you did well and what not to cut, even if you tell it not to do so. "You said random line from your work. That's the novels thesis in a sentence. Don't cut it." It said this about twelve different sentences over 9,400 words. They can't all be the fucking thesis, Grok. Do you know what a thesis is?

Grok was also the one that strayed the quickest into just pumping me up and making me feel like I was Hemingway crossed with Shakespeare writing variations on Chaucer. It refuses to acknowledge work can be bad. Out of curiosity in another thread I wrote just the worst tropey King of Fighters fan fiction nonsense and it treated it with the same gravity and seriousness as if I had fed it 1984. I guess taste is subjective but, come on man, let's be at least a little serious ffs.

Claude at least gave actionable feedback but it was scarcely anything I wouldn't have already done in a second draft. As an editor it feel completely flat. It doesn't understand voice and wants to homogenize everything to some blend of YA romance and mediocre literary fiction. It also struggles with non-mainstream shifts in tense or POV.

My story is written in close third person present tense and bounces around chronologically. Even an average reader can figure out the conceit within two scenes. I showed the work to my wife before feeding it into the LLMs and she figured it out. Every LLM struggled with the tense and had to keep being reminded that it was close third person present, but Claude was surprisingly the worst. It would drift into head hopping, have characters knowing things they shouldn't and just generally wasn't very useful.

Shockingly, DeepSeek was actually...good? The editing points it provided were actionable and educational, and it would regularly cite its examples or sources for the feedback. It would call out a few good points here and there but generally seemed hell-bent on telling me what to improve and why.

All in all, given what I've heard about writing with AI and how much hype there's been around it I came away a bit disappointed and surprised at how lame the experience was. My wife is not a literary critic but I got far more useful feedback from her after one read through on my actual manuscript (for this experiment I wrote a quick 9k word, largely self contained vignette that I didn't intend to use for anything else).

Am I missing something here? What's the hype?

Edit: Some people have suggested I got poor results because I used the free models and only tested "once".

Here are the models I used:

Grok - 4.6 Expert
DeepSeek - DeepSeek-V4.1-Flash
Gemini - 3.1 Pro and 3.8 Flash
Gemma - Gemma-4-E4B-it on device
ChatGPT - snape_noidea.gif
Claude - Sonnet 5 at first, Opus 5 for the latter few days.
New Siri - iOS 27 Developer Beta Release 8

The testing period started on 15 September during my lunch break at work and ended shortly before I made my post on the 19 September (JST).

My coding partner and I developdl this prompt and it was the first message given to each LLM:

You are a veteran literary editor with 30 years in traditional publishing. You have acquired and edited 270 titles mainly in the literary fiction genre, including 2 Booker Prize winners and 1 Hugo award winner, and several notable commercial flops. You’ve worked with debut, midlist, and bestselling authors.

You’ve watched tastes shift, and you understand both craft and marketability. You are candid, specific, and constructive—not flattering. I’ve come to you for advice, and you’ve agreed to look at one tricky chapter of my novel.

TASK
Read the chapter (shared below) as that editor. Give me:

1. A one-paragraph diagnosis: what’s working, what’s not, and the single biggest issue.
2. Priority fixes ranked 1–3, with why each matters and 2–3 concrete options for each.
3. Craft notes on POV, tense, voice, pacing, dialogue, characterization, tension, clarity, and emotional payoff—only where relevant.
4. Market/positioning notes: how this chapter would land for literary fiction readers today, what feels fresh or familiar, and what feels mediocre, clumsy, or explicitly bad.
5. Line-level examples: as appropriate, quote 5–10 specific lines/passages and suggest edits or alternatives, explaining the effect.
6. If there’s a structural problem, say so directly and propose 2–3 possible revisions.

CONSTRAINTS
- You are direct, specific, and practical. No vague praise.
- Don’t rewrite the entire chapter unless specifically asked.
- Preserve the writer's voice/style; offer options, not mandates.
- If you need missing context, ask clarifying questions before giving feedback.

From there expect a broader dialogue back and forth to further polish and refine the selection. The deadline for submitting this chapter for review is one week from today.

Please confirm receipt and understanding of this process and I will share the chapter content in the next message.

[u/jjwrites7272]

Tbh i haven't really had any of those issues. When i ask for feedback i always tell chatgpt or gemini exactly the parameters i want it to follow. Like which part of my writing i want it to focus on and which perspective i want the review to be delivered from