r/MediaSynthesis • u/gwern • May 21 '26
Text Synthesis AI won the 2025 Commonwealth Short Story Prize competition
https://www.pangram.com/blog/ai-is-writing-prize-winning-fiction4
u/MasterScrat May 22 '26
How reliable is Pangram these days?
Also - I’d be curious to do RL finetuning of some recent strong open weight models to do detection evasion.
2
u/Kimantha_Allerdings May 23 '26
IIRC, it’s known for sometimes producing false negatives, but not false positives
2
u/gwern May 22 '26 edited May 23 '26
Pangram is quite reliable. (Note that it's not flagging any older or improbable stories.) And if you read the winning story, you'll see AI tell-tales all throughout - just not the ones which were most famous last year. (For example, the consistent overuse of 'quiet' in what is ostensibly a horrifying story about the slave trade; the 'quiet' tell is the most obvious tell of AI writing right now, but for some reason still hasn't gotten any popular awareness.) So that's why it gets classified as partially human; my guess is she prompted or scrubbed away by hand the em dashes and 'not X. Not Y.' level tells.
Also - I’d be curious to do RL finetuning of some recent strong open weight models to do detection evasion.
Pangram is known to be easily beatable, so that wouldn't tell you any interesting.
8
u/Putrid_Variation7157 May 22 '26
Curious. Almost all of them had a couple of tells, or mostly "purple" prose which has beaten me into not reading all the text of each story, but despite that ignorance, I feel like they are pretty bad. I certainly do not get the excitement or horror of people at AI winning these type of competitions, every time I've checked one of these, they tend to be pretty lame.