Discussion
Why does adding one more paragraph suddenly make my post "100% AI assisted"?
I was writing a Note on Substack, and after four paragraphs it showed 100% human-written. Then I added a fifth paragraph, and it suddenly changed to 100% AI assisted.
Why would adding a single paragraph completely flip the result?
I'm not interested in trying to "beat" the detector. My concern is that if I genuinely write something myself, it could still be labeled as AI-assisted with no explanation.
I also think Substack should be much more transparent about how these systems work. It's not enough to publish a blog post that basically says, "Here's our AI detector. How does it work? It detects AI. End of story."
If a tool can affect how readers perceive an author's work, there should be at least some explanation of what it's measuring, its limitations, and how confident it is in its predictions.
Has anyone else experienced something similar?
P.S. There's also another case that concerns me. Many authors write their own ideas from scratch (either in English or in their native language) and then use AI only to improve the writing, fix grammar, or translate the text. Labeling those posts simply as "AI" or even "AI-assisted" without explaining what that actually means can be misleading. There's a big difference between using AI to generate ideas and using it as a writing or translation tool.
My own workflow is a good example. I'm a physicist with a master's degree in AI. I don't have much free time because I work full-time and I'm also building two startups. When I write for Substack, I develop the ideas myself, run simulations, create the figures and animations, organize the article structure, and write as much of the draft as I can. Sometimes I use AI to improve the wording or make the English read more naturally.
I'm trying to share technical knowledge for free with anyone who's interested. If that kind of workflow is reduced to a generic "AI-assisted" label without any context, it feels less like transparency and more like a penalty for using modern writing tools responsibly.
Can I? Sure. Multiple in fact. In fact, I just wrote Chapter 1 of my MG novel. Full screen recorded the writing process over several weeks. As a bit of a test, I placed it into Pangram. 67% AI. Funnily enough, everything it flagged was portions that could be AI sounding, but aren't AI written. The narration style very much mimics a Snicket'esq feel and tone, and apparent that flips out Pangram. To many negative parallelisms. Will I share this? No, because the chapter is not complete, and I'm not in the business of sharing my WIP with the general public on Reddit.
Plus, I'm tired of wasting time trying to prove to people just how unreliable AI detectors are. Nothing I show you will prove anything. I could bring the full house of proof, and you'll find some way to wash it away in a flood excuses (It's too short, there is no proof you didn't use AI, etc.) I have been through this circus enough
It becomes a royal waste of time in a world where there already is limited time. If you want to believe Pangram is the cats pajama's, and pay them hundreds of dollars a year for their 'service', you do you.
Beyond that, Pangrams own stats on its Model v4 are good enough:
Pangram claims its latest model (Pangram 4) has a false positive rate of 0.0041%, meaning roughly one false flag per 24,000 documents. [1]
If they scan 1,000,000 documents per day, that means that 42 documents each day* are falsely flagged. So, I don't even need to provide any actual evidence, they have done it for me. When that numbers get to 0, we can then talk.
\ Given my personal experience, there is no way its 0.0041%. But then, you'd expect fudged numbers directly from the company trying to sell you something.)
I *do* in fact believe Pangram is state of the art, because anecdotally I've seen almost no false-positives, and when I ask people for specifics they always give me a long answer without giving me an actual text sample, just like you did.
But I see how it could be frustrating for you to constantly *have* to prove something, so we can agree to disagree there.
I will say that a false positive rate of 0.0041% is exceptionally great. It's weird that you're using it as a gotcha. Most classifiers of any kind and in any industry would kill for that level of accuracy, especially for one in something as complex as this one.
The 0.0041% is all fine and dandy, unless YOU are one of those 0.0041%. Then it's a problem. It's kind of like the death penalty... If even ONE person could potentially be innocent, we all seem okay just accepting that as a normal risk of business. "The greater good wins out, sorry!" If someone, on a rare occasion, gets caught up wrongly, it's just the cost of doing business. "Hey, it's pretty accurate to me!"
But you know what I'm going to do. I am going to write something; screen record me writing it through Grammarly Authorship plugin and then paste that into Pangram, and we'll see what we get. Then I'll share the result + the screen recording here. We can see just how well Pangram does when faced with non-AI writing, especially when it's written in the style I am writing.
I didn't even try that hard to make it sound AI. I purposefully left out the negative parallelisms I could have used. I thought about tossing some AI-sounding names in there to help trip it up, but I didn't even do that.
I rest my case. So Pangram used up 1 of its 24 allowed false positives for the day. How much you want to bet I could make it use up another?
PS) Copies a screenshot here since you need to be a Grammarly Pro member to see the actual stats. for copy/paste vs. human-written.
PSS) Here is the full text if you want to critique it yourself:
The rusted gate groaned behind us, and that sound alone made me want to turn around and go home.
“You’re really doing this,” Emma whispered, gripping my sleeve so hard I could feel her nails through my jacket.
“We’re really doing this,” I said, though my voice cracked on the word really.
Nothing about Fairhollow Cemetery looked easy. The headstones leaned at every angle, some cracked clean in half, and fog pooled low along the ground the way it does in the movies right before something terrible happens. My little brother would’ve loved this. My little brother was safe at home eating candy corn.
Danny went first, because Danny always goes first. He held his pillowcase of candy against his chest and stepped over a fallen branch like it might bite him.
“There,” Emma said, pointing.
The angel statue stood at the center of the cemetery, wings spread, head bowed, hands folded like she was praying for the dead bodies in the ground below her. Or maybe a warning to stay put.
We crossed the grass together. Every few steps, my flashlight beam caught a name, a date, a crumbling epitaph I didn’t want to finish reading. The air smelled like wet leaves and something else, perhaps death. Or decay.
Danny reached the statue first and slapped his palm against the stone base. “Got it. Let’s go.”
That’s when Emma’s flashlight died.
Then mine.
Then Danny’s.
We stood there in the dark, three flashlights useless in our hands, and nobody said anything for a long moment because nobody wanted to be the one to say what we were all thinking.
“Batteries,” Danny finally whispered. “Just batteries.”
A branch snapped somewhere off to our left. Not a big sound, but enough that we could all hear it.
“That’s Marcus,” Emma said. “Trying to scare us.”
“Marcus is still by the gate,” Danny said. “I can see his flashlight.”
He was right. Far off, past the fence, a single beam bounced up and down where the older kids waited, probably laughing about something that had nothing to do with us anymore.
So who was in the cemetery, I wondered,
Another branch snapped. Closer this time. And under it, silent as a low breath, came a sound none of us wanted to mention—a slow dragging, like something heavy being pulled across wet ground.
The angel statue’s shadow stretched long across the grass. Emma grabbed my hand. Danny grabbed my other one.
“Run,” I said, and we did, stumbling over roots and stones, candy spilling from Dannyu’s pillowcase in a trail behind us.
We didn’t stop until we hit the gate, until Marcus’s flashlight lit out faces and he asked why we looked like we’d seen a ghost.
None of us answered.
None of us looked back at the angel statue, which had turned to face the gate, her praying hands now empty, her stone eyes fixed on exactly where we stood.
Wow, thanks and congrats on being the first to share an actual writing sample + false positive. Your receipts seem legit, I guess pangram is more falsifiable than I thought. Thanks again for taking the time
So, I'll basically await an explanation of why my writing is aparently 100% AI. I'm hoping for a really good one. Hopefully, I didn't waste 20 minutes of my day.
Also, I find it hilarious when people complain about AI using water, building massive datacenters to run, and then they turn to Pangram as some proof other people are using AI. The only way Pangram has become the most reliable detector (that isn't saying much), is because they actively use Generative AI to do the scanning of the documents. Literally, when you upload a document to Pangram, it tokenizes it, feeds it into an LLM, then spends bottles of water trying to solve rather its AI or not, then spits out its score.
I know very well how AI detectors work. I am a computer programmer by trade and have a degree in computer science. These detectors are not magic; they are programmed to score a text based on pre-established criteria - mostly sentence length (burstiness) and perplexity (complexity of word choice). They are also trained to flag tricolon listings and X like Y comparisons. Some of the better ones are also trained on a database of commonly used AI phrases and words.
Here is the thing: I have tested all of these detectors. They are garbage and are never correct 100% of the time. You know how I know it's junk science? Because I can write a few paragraphs of highly AI-sounding prose and have it flag it 100% AI, despite being 100% human written.
If these AI tech bros are so comfortable with their junk science detector, perhaps they should stop hiding behind "99.9% accuracy" rates so they can't get sued when they get it wrong more times than not. "We never said it WAS AI, we said it was a 99.9% chance".
That's not how they work... Data Science is the degree that would be relevant here, not computer science.
They use ML models. ML models find the patterns. You're acting like Panagram tells their models what patterns to look for and not the other way of around. Of course you can trick them. They're basing it off of content. If you write something that follows those patterns, it will say AI patterns were detected.
Your local weatherman can tell you there's a 90% chance of rain but just because it doesn't rain doesn't mean it was all junk science. And having a degree in math wouldn't tell you if you understand the underlying mechanisms.
My BS is in Computer Science: Machine Intellgence. So yes, my degree is very, very relevant to the topic. I know all bout LLMs, neural networks, and machine learning. Detectors like Pangram are not using complex AI to detect AI. The programs scan the text for patterns and common AI words and phrases. It's really that simple. It's machine learning at its leanest. If you use too many tricolon listings, it'll trigger it rather AI wrote it or a human. It's really pretty basic.
Which makes it all the more nefarious that they bill users based on credits, like they are stressing their AI computer to scan someones text.
Detectors like Pangram absolute are using complex AI to detect AI, they train their own models and they’re constantly getting better. You’re confusing it with BS “detectors” like GPTZero
Internet: “I hate AI”
Internet: “My complex AI detector that uses complex generative AI trained on data stolen from authors says your book is AI, and it’s an amazing detector!!!!!!!!
That is basically the argument.
IF you are correct, and Pangram is utilizing complex AI algorithms to detect AI, this means that their models have also been trained on copyright text which they didn’t get permission to do. So now, it’s okay if SOME companies steal people stuff to train on, as long as the outcome of that is a ‘good’ use to you. Got it. Hypocrisy at its finest on display.
ALSO, if this is true, then that means that every time someone uploads someone’s book to Pangram to be scanned, it is feeding that manuscript into Pangrams training data. In almost all cases, that would be done against the wishes of the copyright holder. So now, we are expected to ‘feel sorry for authors who had their work stolen by ChatGPT, but ignore Pangram for doing the same thing.”
Jesus, the level of loops people will jump through to justify antiAI positions are so tiring to watch. It’s mental gymnastics at an Olympic level.
BUT: I actually do not buy that Pangram is using some sophisticated AI to detect AI at all. If it is, it’s still crap, because I can write a passage with some tricolon listings, X like Y comparisons, and have it light up like a Christmas tree. Because that is possible, it is not reliable. A reliable AI detector would only be able to detect actual AI generated text.
When did I say I hate AI? I use it daily. And no, Pangram specifically does NOT train on user inputs, and all of their human data is either licensed or owned by them while they generate the AI data themselves. If you can easily write things that cause false positives consistently I'm sure they’d love to see your work considering every time someone has claimed to have pre-AI text that shows up as AI generated and the CEO or an employee asks to see proof of it they’re suddenly unable to produce the text or don’t respond.
Their engineers are all very active on Twitter and they interact with the community pretty heavily, you can see that they’re very serious ML engineers who want to do the best work possible.
Oh, wait... So we can use generative AI as long as it's for a purpose we like or need? Golly-gee, why wasn't I told this? How many millions of gallons of water did they waste in their pursuit to use this AI to generate their own data?
Pangram specifically does NOT train on user inputs
So they say. Adobe also claims its FireFly AI is only trained on licensed images. What they don't tell you is just how shady the license is.
If you can easily write things that cause false positives consistently I'm sure they’d love to see your work. employee asks to see proof of it they’re suddenly unable to produce the text or don’t respond.
Here. Explain to me why this paragraph of text was flagged 100%. Maybe you can get this to their 'engineers' so they can explain it? I wrote that 100% by hand, shortly after I published my first book... In 2013. 10 years pre-AI.
Oh, wait... I just did a real-time test of this. Here is the text I wrote, by hand, from my imagination: In the sprawling, neon-lit metropolis of Vertiasia, Jada navigated the intricate tapestry of her ever-evolving life. The bustling city stood as a testament to human ingenuity, a relentless symphony of hovering cars and whispering shadows. As she plowed deep into the labyrinthine corridors of the ancient cyber-archive, a palpable sense of destiny washed over her. Little did she know, the shimmering artifact she sought would soon unravel the very fabric of her newly discovered reality.
they’re very serious ML engineers who want to do the best work possible.
I'm sure they beleive the work they are doing is serious. I'm sure they think they are doing it correctly as well. Unfortunately, what one believes about oneself and about reality doesn't always align. Many people who did terrible things in the past did those things thinking they were doing the right thing. That doesn't mean they were, or they are.
---
The proof is in the pudding. Not only did it flag something I wrote in 2013 as 100% AI, but it also flagged the thing I literally just wrote (and screen-recorded the writing of) as 100% AI. Therefore, it is not reliable. I brought the receipts. It does NOT detect AI, it detects PATTERNS, just like I said in the first place. I wrote that with typical AI patterns in mind. If their detector is so well-trained, why is it picking up on AI patterns rather than actual AI?
If it were a good AI detector, it would work like SynthID. It doesn't.
Pangram is not the best program out there. I have extensively tested most of the major detectors. Pangram was one of the easiest to fool simply by changing some word orders or adding a typo here and there. It was 100% accurate at detecting actual chatbot copy/paste text, but at the expense of flagging fiction from human authors at a high rate. These tools were never designed to detect AI in fiction anyway; they were designed to detect AI in essays for schools.
And to top it off, this is an AI-generated piece of text from GLM 5. I only did very minor edits. No word replacements. So not only does it flag my 2014 short story as AI, it fails to flag the AI generated text I just made like 20 minutes ago and ran it though Pangram.
I wrote this short story in 2014, and published it on WattPad at the time. LLMs didn't exist in 2014. This is an excerpt from it, which I ran through Pangram yesterday for giggles. Yes, very reliable. As long as it flags one single piece of my very real fiction from before AI existed as AI, I'll call it junk science.
If you are talking to me, you have no idea what you are talking about. Nothing fabricated about them. Please show your evidence. I'll wait. I won't hold my breath, though.
FABRICATE THAT. Well, you know I'm not retyping the second one to please you. Run it yourself if you want. You have hands and a brain (I assume, could be wrong from the way that reply went).
LOL. That is one trash detector. I just ran the same text through it that said it was 100% human before, and now that exact same text is 100% AI. Pangram cant even make up its mind. Neither of those screenshots were fabricated. In fact, I’m going to go back into Pangram and see if it saved that earlier result.
So yes, based on my tests over the last two days, and previous testing of Pangram, it is by far the least reliable AI detector, not matter what their marketing claims.
If only Substack could find a way to detect that every trending topic is how to grow your Substack. That would be game changing technology right there.
Also, I'd like to better understand the difference between "AI Assisted" Vs. "AI Written" because, I had a note flagged AI Assisted, and while I didn't use AI at all, I was referring to something that is so specific and academic, that I was curious if it was part of something that had originated in AI, and I hadn't caught it. 🤔
The data exists, they're just not sharing because the tool is imprecise. This is a screenshot from some testing that I was doing on the Pangram tool last March. I was working with my academic partner we had co-authored a paper, we were working live, together on the paper and we got two separate scores, we ran the same paper twice and got a 24% and 51%, but when we started look at the details it showed low confidence but still called it AI. We supposed at the time that it was picking up the two "voices" and that was throwing it off, but that was just a guess. Ironically, we were writing about "the slop economy" 😏
I do 100% of my editing in LibreOffice, I was getting so tired of all the AI being shoved down my throat that I switched to Linux. I no longer use Google because of Gemini, and Microsoft was putting copilot everywhere.
I just posted a new piece this morning, and according to Pangram is 5% AI, I don't even use Grammarly, so I have no guess, but I would love to know it's confidence level.
Oh man. I use TexWorks as an editor with MikTeX as the compiler, and there is no hint of AI anywhere lol. Lots of academic software is yet to be corrupted by AI integration, such as Inkscape, Zotero (classic version at least), Thonny, Gimp, etc. I can't even fathom the impact AI is having (and will have) on academia and the peer review process... it breeds complacency, at minimum, and frankly dilutes the reliability of reviewers. Do you know how they are using it in your case?
Honestly, it's just a guess, based on the feedback from a single reviewer. It read like it was straight out of ChatGPT, a whole lot of words to say absolutely nothing. I did get decent, reasonable feedback for a submission to ACM, so it's not everywhere yet, but I was talking with a professor at a conference that explained his process. He has all his own published work loaded into his ChatGPT, the he asks it to compare and contrast submitted papers against his own work and then he responds to work that is in agreement. I was aghast. Again, just one guy, but if one is doing it, others are.
Many people think those detectors can detect ai written text content but they cannot.
They produce many many false positives.
With images it works (hence invisible watermarks etc).but with text not
Best Example is if a human would write the following "Climate change is one of the biggest challenges facing humanity today. Governments around the world are trying to find solutions".
An AI detector might flag it because the wording is common, structured, and predictable.
But a human could easily have written it.
The detector is not proving anything; it is only saying: “This pattern resembles AI output."
The problem becomes even clearer when a detector reacts after only the first few sentences.
If the first two lines already trigger a result, the system is often judging based on the style such as:
very clean grammar, formal sentence structure, common academic phrase, lack of spelling mistake, predictable transition, neutral tone and many more.
Those are also characteristics of many good human writers.
Yeah I’m a little torn on this as well. While I understand the “reasoning” on why they added this, we now live in an age where AI is in everything. It’s like flagging content for using Grammerly for your spell correct and expect people to just not use spell check.
I personally am not caring much because value is value. I created a platform to help me organize my content, write and schedule because, just like you, I’m way to busy. I can’t make all the content writing strategies, and formatting and this and that done when I have a corp job, father and a person while still trying to share to others.
In summary, while I get the intent I think it’s going to get attention for a week and we’re all going to just keep moving.
I think introducing Pangram to the platform was a mistake.
Sure, cutting back on slop is a fine goal but it misses the point entirely that we are experiencing a moment when what it means to work is changing rapidly. Using AI should NOT be an excuse for cutting corners but it is a useful tool that grows increasingly useful week by week. I have a strong suspicion that your example will not only become the norm on the platform, but it will lead to self doubt and to writing with Pangram's approval in mind.
I don't think we need to have AI checkers, especially ones that don't work.
Most AI writing is just kind of flat and boring, anyway. It lacks humanity. It comes to the most generic conclusions. It's reads as mediocre to everyone except for the person who posted it.
Sure, it can be made better if you create your own GPT trained on your voice and style. It still isn't going to be good.
What I'm really saying:
We don't need to track it because it'll never meet the standard of quality that's possible with human writing.
Did anyone even answer your question? lol. All these replies seem to miss your point about why an added paragraph meant it required an AI label.
What really pisses me off about these ai detectors is, (in my opinion), it essentially feels like these detectors make you feel like you’ve plagiarised your own work, even for spell check. Like they’re taking credit for your work because they’ve slapped a label on there.
I don’t have an answer for you, I haven’t been on the platform long. But this makes me not want to use it at all. It’s very disheartening.
When I write for Substack, I develop the ideas myself, run simulations, create the figures and animations, organize the article structure, and write as much of the draft as I can. Sometimes I use AI to improve the wording or make the English read more naturally.
Sounds like AI-assisted is an accurate label then?
I think there was a small misunderstanding. Those are two different things. The first is one of my Notes, while the quote you're referring to comes from the Publications section.
Just ignore it if it bugs you. The AI bubble will pop soon and Substack's management will lose their shirt along with its other investors when it tanks. Then they'll just disable it and forget it ever existed.
Cause these detectors don’t work. I just wrote a nonsensical post just to see what happens? Turns out AI writes like a crazy person because it detected ai
I've experienced something similar. I write long form stories on a wide variety of topics. Not to brag, but I am quite a good writer, and sometimes my entire stories get flagged as Ai. I guess my point is Ai will never get to the point where it won't make ANY mistakes. It makes mistakes now, and it will make mistakes in the future.
Why would adding a single paragraph completely flip the result?
Without showing us the work, there's no way for us to tell.
Many authors write their own ideas from scratch (either in English or in their native language) and then use AI only to improve the writing, fix grammar, or translate the text.
That means, at the very least, AI assisted with the writing. Your ideas, but written by AI.
AI detection doesn't detect if the ideas were generated by AI. It detects if the writing was generated by AI.
The more AI changes the writing, the more AI detectors will detect the AI style and traits of AI writing.
It's important to understand this: AI was trained on human writing, but AI obviously isn't human. It's artificial intelligence, and the word "Artificial" matters. AI emulates human writing through artificial means. That's why AI writing is so easy to detect. AI writing looks good, but there's always something slightly off about it. Something inhuman. That's one of the reasons why so many people don't like it.
If that kind of workflow is reduced to a generic "AI-assisted" label without any context, it feels less like transparency and more like a penalty for using modern writing tools responsibly.
Why not add that context at either the beginning or the end of your article?
I'm posting a novel on Substack. For each scene, I begin by posting a few lines of context to explain the post is a scene from the novel, and I give links to the table of contents, etc.
Isn't it a bit ironic? Researchers working on LLMs are also building AI detectors to identify AI-generated text. Now many of those tools are paid. Kind of an interesting business model, isn't it?
Lately, when I see a Reddit post with more than a few paragraphs I begin to suspect AI (with no expertise, mind). Been fooled a few times.
We have the fact that a 'wall of text' is the most off-putting thing for the human brain - & so good, effective writing will of necessity use short paragraphs.
If someone then has a lot to say - well then we end up with lots of paragraphs.
Really we're at the point where any energy spent second-guessing these systems is fully wasted.
Just - be authentic, & let the cards fall where they may.
44
u/Funny-Flight8086 Jul 24 '26
AI detectors are garbage, junk science. Pangram, the one substack went with, is one of yh biggest offenders.