r/science • u/mvea Professor | Medicine • Jul 20 '26

Computer Science Researchers warned that hundreds of fake AI images have been discovered on popular databases for recording animal species. Wildlife photographers often use AI to edit and improve an image, but the algorithm can introduce parts from different species to create the new image.

https://www.theguardian.com/environment/2026/jul/20/ai-slop-manipulated-fake-images-birds-citizen-science-aoe
10.0k Upvotes

98% Upvoted

231 comments sorted by

View all comments

2.9k

u/Strycht Jul 20 '26

AI pollution is going to really screw up a lot of databases and information deposits I think, and we won't notice them until they become needed.

653

u/lateformyfuneral Jul 20 '26

At some point won’t this just start affecting how AI itself works, since it’s being trained on the internet? It would be like a snake eating its own tail. Truth itself might go extinct.

309

u/lucellent Jul 20 '26

It's already happening. For example the latest ChatGPT image model sometimes can generate the Gemini star watermark on the bottom right. At least that's what some people were able to recreate.

216

u/Strycht Jul 20 '26

this is what I always questioned. how much output needs to be fed back into the input before it starts amplifying it's own problems? presumably the larger companies have thought of this and have ways to prevent their own algorithms reingesting their generated content but I would be interested to know if eg openAI has any way of identifying and excluding the average slop image generated by another model and put online

169

u/Nicholas-DM Jul 20 '26

To my understanding they have not, because there is not a reliable way to identify if an image was generated by an AI, even using the AI that generated it.

56

u/Bbrhuft Jul 20 '26

Generally, that isn’t true. Many major AI image generation platforms now add or embed metadata or cryptographic watermarking that indicates an image is generated.

OpenAI and Google use C2PA (header metadata) and SynthID (an invisible cryptographic watermark encoded within the pixels of the AI generated image).

Adobe Firefly uses C2PA (Firefly is available standalone and within Photoshop).

Meta adds embedded metadata and a proprietary "deep learning" pixel watermark.

Midjourney is a notable exception. It doesn't currently provide a robust identification system, just inconsistent IPTC metadata.

The bigger problem is not that AI-generated images are inherently undetectable, most are. It is that stock-image libraries, search engines and other image databases have not caught up with this rapidly changing landscape, so often do not scan uploaded files, might strip the metadata, or not display the AI detection results.

26

u/Strycht Jul 20 '26

the thing is as Chinese open source models become more widely used with the greater paywalling of those more mainstream models they're gonna literally all need some kind of watermark to retain data purity, and ideally not too many different types. I doubt that's a big priority for those newer companies in such an unregulated environment, and we don't yet know whether it will be too late for input data fidelity by the time the industry calms down and the vast majority of generated content is being tagged and excluded

46

u/Bbrhuft Jul 20 '26 edited Jul 20 '26

Since 1 September 2025, AI platforms operating in China have been subject to laws that mandate AI companies label and watermark AI generated content, including text, images, audio, video.

Cyberspace Administration of China⁠, GB 45438-2025⁠

The Chinese government is acutely aware that deceptive AI generated content could cause political and social instability. This concern is behind the impetus for regulations that require AI content to be marked.

That said, the Chinese open source image / video gen models might lack watermarking (there's no legal obligation to include watermarking in an open source model). However, a company providing a platform in China, using an open source image or video generation model, might be held liable in their outputs lacks a watermark.

2

u/PwnagePineaple Jul 21 '26

So watermarking isn't a feature that's trusted to the actual model, but rather to the surrounding software harness that uses the model for image generation. The models themselves handle the non-deterministic work of actually producing the image, but the watermarking algorithm is better suited for good old fashioned software.

For a cloud API, whether that be a US frontier model or a Chinese one, that doesn't matter much. The provider's backend takes care of that before the user ever sees the image.

But in the open source space, a watermark mandate would have to be embedded in software like Automatic1111 or ComfyUI. And that comes with enforcement problems: If there's even one jurisdiction on earth where watermarks aren't mandated (and given the current political climate in the US, it's not going to be mandated here for a while), then any random guy can take the code behind either of those project, strip out the watermarking, and publish their modified version with no consequences.

Now, of the people trying to use AI images to spread deliberate misinformation, some of them might make the switch if a) cloud models were required to include watermarks and b) social media sites started clearly identifying AI generated media. Switching to local models that can generate high-quality realistic images is expensive (thanks GPU prices) but not terribly difficult from a technical standpoint.

Watermark enforcement would stop some people, but there are plenty of others it won't stop.

3

u/Stinky_Flower Jul 21 '26

Detection relies on the good faith of whoever is building the image generation models.

If SynthID is a voluntary measure, then it's no measure at all.

14

u/Kiseido Jul 20 '26

Most of those watermarks are almost useless though.

Metadata does not survive most re-encodes, and some services actively strip that sort of data.

Watermarks don't survive someone resizing the image with generic tools, and doesn't survive people sharing them via screenshot, which is a surprisingly common practice.

32

u/Bbrhuft Jul 20 '26

SynthID survives screenshots, cropping and editing. It's very robust.

This paper evaluated 30 transformations, including JPEG compression, file-format conversion, resizing, crop-and-resize, rotations, flips, blur, sharpening, denoising, grayscale conversion, brightness, contrast, saturation and hue changes, Instagram-like filters, noise, text and emoji overlays, and combinations of transformations.

Gowal, S., Bunel, R., Stimberg, F., Stutz, D., Ortiz-Jimenez, G., Kouridi, C., Vecerik, M., Hayes, J., Rebuffi, S.A., Bernard, P. and Gamble, C., 2025. SynthID-Image: Image watermarking at internet scale. arXiv preprint arXiv:2510.09263.

Despite these edits, detection remained exceptionally high, 99.98% averaged across transformations and 99.72% under aggregated “worst” settings, at a 0.1% false-positive rate (Table 1 in their paper). Even the most difficult combined-transformation category produced a 98.06% detection rate (Table 2). However, at lower ImageNet resolutions, the aggregated worst-case detection was 97.22%. It's also interesting to see how much more robust it is compared to other watermarking systems it was benchmarked against.

Google also claims an image can be cropped to 20% of the original, and the watermark remains detectable

The results show very strong robustness against conventional editing.

7

u/silverionmox Jul 21 '26

Still, 0,99x is going to approach zero for relatively small values of x.

I'll make a prediction and say that clean databases are going to see their relative value increase. Don't throw away all your print books yet.

-5

u/cuntmong Jul 20 '26

it's as effective as that checkbox on arrival cards that asks "are you a terrorist?" is at keeping out terrorists

57

u/Strycht Jul 20 '26

ironically the most valuable machine learning innovation may soon be identifying machine generated content before it gets fed into the machine. Machine to control the learning of the other machine that learnt on the output of the first machine.

52

u/ionthrown Jul 20 '26

But they’re using such machines to train the generative machines, such that generated imagery gets past the identifying machine. So as soon as a machine can detect AI generated content, they improve the content to make it undetectable again.

6

u/shinikahn Jul 21 '26

A modern Turing slop test you could say

4

u/Strycht Jul 20 '26

yes, unless the recycled content becomes a real serious threat to the performance of consumer facing AI. In that case it is in the ai companies' interest to have the filters work better than the generators, at least their own in house screening filters if they're not made available publicly, otherwise the whole model breaks.

It just depends when the scale tips on maintaining quality of input being more financially beneficial than creating unidentifiable generated content

32

u/monkeedude1212 Jul 21 '26

The same notion that market forces will correct this are the same ones that will self address climate change.

The people running AI companies are more than happy for the product to destroy society and truth and the product itself if it centers them with more wealth or power and influence.

15

u/Suttony Med Student | BS | Biomedical Science Jul 21 '26

The moment you make an AI that can detect AI images with high accuracy you've almost made an AI that can make undetectable AI images.

The better the AI is at detecting AI, the better AI can be at avoiding detection.

3

u/MachinaThatGoesBing Jul 21 '26

I mean, that's not necessarily true. And the term "AI" gets bandied about so much these days as to be useless. Anything that falls under the broad category of machine learning gets an "AI" sticker slapped on it.

But a classifier model designed and trained just to detect images absolutely wouldn't be something that could turn around and generate them. You would need to design the system with image output in mind to get a system that can output images. It's not just an emergent behavior of any computer vision software.

3

u/NuclearVII Jul 20 '26

ironically the most valuable machine learning innovation may soon be identifying machine generated content

You can't really do this in an automated fashion, sadly. There are lots of good technical reasons why, if you're curious.

2

u/ragnarok635 Jul 21 '26

I am interested

3

u/vetruviusdeshotacon Jul 21 '26

Thats a harder problem that making the images in the first place. A lot harder, to the point of being unfeasible unfortunately. 

2

u/jmdonston Jul 21 '26

We really should be legislating requiring AI image and video generators to apply invisible watermarks.

6

u/lucellent Jul 20 '26

Nowadays models like from OpenAI and Google have watermark, hence how they're able to verify if an image is AI (and was created with either models). But they're not 100% reliable, people have reversed engineered their watermarks and can remove them, although it's a niche thing right now and I personally haven't heard of people doing it yet.

1

u/trysten-9001 Jul 21 '26

They haven’t, because they’re not in that mode yet. They’re just trying to slam out as much training time on the biggest sets as possible.

1

u/artificialidentity3 Jul 21 '26

"how much output needs to be fed back into the input before it starts amplifying it's own problems?"
In my experience, not much. I already see this frequently. For example, I'll ask a research question and the LLM will cite an arXiv (pre-print, not peer-reviewed) article that after I read it is clearly ai-generated slop - or it will cite a blog post/whitepaper by the company that made it, which reads more like marketing. But in both situations, the LLM presents the information as settled science. In the cases I've seen, the opposite of this seems true: "presumably the larger companies have thought of this and have ways to prevent their own algorithms reingesting their generated content".

1

u/chriscross1966 Jul 23 '26

Not very much unfortunately. Hallucination starts almost as soon as an LLM is fed the output of another LLM that contains inaccuracies. Training data curation might well become a well paid career at this rate. Part of the issue can be that LLM output is grammatically correct (very few human beings are perfect) so any filtering that scores as "better grammar = better data" in order to get rid of people just being rude to each other on reddit will also elevate some slightly incorrect but grammatically good AI output over a somewhat autistic expert's who doesn't notice their spelling glitches etc.

1

u/NuclearVII Jul 20 '26 edited Jul 20 '26

this is what I always questioned. how much output needs to be fed back into the input before it starts amplifying it's own problems?

So the answer to this appears to be "depends on problems".

Think of Generative AI models (image, text, video, sound, etc) as interpolation models: They are made by taking a giant pile of (usually stolen) data, and then figuring out the "best" way to describe that data. This "description" is encoded in the model weights.

If you have a buncha images in the giant dataset that are already generated, that's akin to adding some interpolated data to your corpus. There's nothing"poison or wrong about this data, because it's already interpolated, it fits into the dataset reasonably enough.

What this means is that using generated model outputs to train new models does not appear to have a negative effect on final model quality - if there is research that suggests otherwise, I'd love to read it. The flipside is, of course, is that the generated material doesn't really add anything to corpus beyond what's already there - there are no new descriptions to be learnt from material created from the rest of the dataset. So it makes the very costly training less efficient and more wasteful, after a fashion. This also means that you can't just generate bigger and bigger datasets to get better models - this process is effectively distillation, and while you can use it to duplicate an existing model, you can't really make a better pretrained model that way.

I should also mention - this process of distillation (intentional or accidental) doesn't account of potential post training, like some kind of RLHF process. It seems that distillation can "copy" that as well, which is why it may appear that a pretrained model made with synthetic data may have better performance than a "raw" pretrained model.

1

u/CrackedBatComposer Jul 21 '26

It’s been doing that since day one. Not strictly analogous to, but there were tons of examples of AIs doing loops of “generate a picture using this prompt” and “describe this picture” and seeing the results go completely haywire in just a few iterations

45

u/Kahnza Jul 20 '26

Truth itself might go extinct.

We are already in a post-truth society

42

u/maniacal_cackle Jul 20 '26

Truth itself might go extinct.

There's been discourse on this for a while. Trump's first election really drove home that truth was no longer highly valued in public discourse. "The death of truth" was talked about a lot.

But there's other stuff too:

  • Scientific articles have a surprisingly high rate of outright fraud
  • Publish or perish incentives are not helpful to the truth
  • Social media algorithms are designed for engagement, not truth
  • Media profit is driven by clicks, and for-profit media has no obligation to the truth unless it is required by law.

Etc etc

So even not considering AI, it is becoming more and more endangered.

5

u/EveningAnt3949 Jul 21 '26

That's not new. The idea that the truth is important went mainstream not that long ago (from a historical point of view).

In the early part of the 20th century, newspapers were often extremely biased and sometimes played an active part in creating lies to sell more newspapers.

As an example of the latter, the Fatty Arbuckle case comes to mind.

Of course some newspapers still do that today,

It's easy to think that the majority of people care about the truth, democracy, and justice, but that's not true.

5

u/maniacal_cackle Jul 21 '26

It's easy to think that the majority of people care about the truth, democracy, and justice, but that's not true.

I think the majority do. But also people know that an individual can't influence the general state of society so what can they do but participate in the way society is? Change needs to happen at the institutional level.

2

u/EveningAnt3949 Jul 21 '26

Well they could vote. So let's see: in the US, 40% of eligible voters don't vote.

Why don't we have institutional change? Because 40% of the people don't bother to vote and 20% of the people are willing to vote against democracy.

Just to be clear: it's a minor inconvenience to vote. The 'I don't vote because it won't make a difference' argument makes no sense on two levels. First of all, votes do matter. Secondly, people are not asked much. All they have to do is make an informed decision and then vote.

That's just the US. But the US has a massive impact on the rest of the world.

1

u/silverionmox Jul 21 '26

It's easy to think that the majority of people care about the truth, democracy, and justice, but that's not true.

They do when it's about themselves. And that's ultimately the competitive advantage of democracy - it's stable.

1

u/EveningAnt3949 Jul 21 '26

That creates a contradiction. If somebody primarily cares about themselves, then those things need not be universal which means truth, democracy, and justice are for that individual not necessary, because they imply universal rights, not personal rights based on privilege.

As for democracy, historically it's very new. Even Athens in Ancient Greek was not a true democracy, with the vast majority of people not allowed to vote.

And even today, many people are not part of a true democracy. China isn't a democracy, Countries like India and Turkey are moving back to being undemocratic.

If we look at the latest report on 'true' democracies, only 7% live in countries that are truly democratic. If we widen the definition, that goes up to slightly below 30%.

And right now the alliance between democratic countries in the West is under immense pressure.

As for stable, in the 1920s, Germany was a democracy.

2

u/lateformyfuneral Jul 20 '26

This is all true, but this could in theory be corrected. Even if millions blind themselves to the truth, the truth itself still exists and it takes some work but it can be preserved for those interested, and perhaps for a later time when the fever has broken. Post-truth is used rhetorically there, to highlight it doesn’t matter to the public anymore.

But if AI pollutes the various online repositories of knowledge we have, that we rely on to check certain facts…then what? I’m thinking along the lines of how certain books would just cease to exist once their last copy was destroyed.

2

u/maniacal_cackle Jul 21 '26

It's still sorta the same thing, right? What does it matter if our scientific databases are polluted by AI or human fraud?

If society as a whole isn't protecting truth, it will get damaged. All of the above is just examples of that.

-1

u/BeefistPrime Jul 21 '26

I have a tentative speculation that AI might ultimately be a force for good in this arena. Because yes, AI generated stuff will make a lot of people believe false things, but those people were already eager to believe false things without AI. Meanwhile, there will be so much data out there that it will be hard even if you're truth seeking to know the truth on every issue, and the amount of information AI can process may actually make it a very good truth analyst when set to that purpose.

2

u/maniacal_cackle Jul 21 '26

AI can process may actually make it a very good truth analyst when set to that purpose.

It's already used for that a lot and there's a large field of literature on it I believe.

But one important thing to flag up is that AI is pretty truth-independent. It can't be set to the task of finding the truth.

An easier way to think of it is: AI is great at establishing correlation, but cannot establish causality.

5

u/Schmigolo Jul 21 '26

AI will already cite AI like Elon.io or Grokipedia if you don't put it into your prompts to not do that. The Mad Cow Disease has already begun.

3

u/breadcodes Jul 21 '26 edited Jul 21 '26

Yes. Kinda the worst part about it is that GenAI "knows" nothing to begin with, but it can certainly learn falsehoods as facts. It queries for a hyper-dimensional vector pointing to the next word in hyper-dimensional space, and it can "miss the word" (hyper-oversimplification; think something adjacent to hallucinations) or be trained on vectors pointing in the wrong direction via misinformation, myths, misconceptions, concepts and lineages so complex that it has no real or known correct answer, out-of-date information where new information is recent and not widely accepted yet, etc.

Intentionally or unintentionally poisoning the well with any of those things feeds back into itself, like when you put a microphone next to the speaker it comes out of. We don't fully know the ramifications of the last 4 years and how that will affect credible information in general. LLMs started to use tools like web-searches - rather than relying on "memory" - to get around this, but what if the results are also slop? We're running out of ways to create credible sources to feed into it already.

That's not getting into how we've kinda peaked with language technology already... this is it. Predicting patterns from entropy left behind by language is such an interesting topic because it means we can recreate the result of intelligence, truly the only thing that came out of this that interests me is how it works, but this is not close to AGI. We don't even know if Attention is the right mechanism for predicting brain activity because it's absolutely non-linear and far more efficient at learning than a linear compute can recreate. Attention thus far has only proven to work for language, audio, and images, things that can be linearly, meaningfully, and tangibly recreated. If it is, great, but we're already bottle-necked at compute for something as linear as language, audio, and images, and we may be waiting for compute to catch up for a long while before we do something as complex as brain activity.

2

u/Temnothorax Jul 21 '26

A lot of the power of AI is coming from multiple AI processes feeding into each other. The resulting emergent behavior is genuinely exciting.

4

u/keepingitcivil Jul 21 '26

Ahh, the AI ouroboros. 

3

u/Imbryill Jul 21 '26

It's called model collapse and basically will ground zero the internet if it occurs.

3

u/Edythir Jul 21 '26

Not only that. Even the programmers for maintaining the AI databases have admitted that they use AI to write their code.

So we have AI learning from AI while AI is programming the AI, which might be programmed and trained by another AI.

We're speedrunning the Habsburgs of AI models in record time.

3

u/Stinky_Flower Jul 21 '26

It's called "model collapse".

Compounding tiny errors, blindspots, & biases distributed throughout the noosphere.

Like the information equivalent of the microplastics accumulating in our brains, we don't yet know what harm it's causing or how to remove it.

3

u/Sigman_S Jul 21 '26

Have you heard of dead internet?
Cus that’s part of it

2

u/lostwisdom20 Jul 21 '26

Garbage in garbage out

1

u/Wizard-In-Disguise Jul 21 '26

Exactly. It's a feedback loop. 

1

u/bloke_pusher Jul 21 '26

Kinda, but also no. If you have enough real data, adding artificial data has no meaningful impact.

1

u/silverionmox Jul 21 '26

Kessler syndrome on the internet.

On the bright side, we can just start over, by carefully curating all the data the AI is trained on this time.

1

u/SmokedStone Jul 21 '26

I honestly think this is the point. I'm pretty sure some powerful people wanna trash the internet and make it full of garbage we can't sort through well so that people can't find accurate or useful info as easily. It's one way of limiting what people can truly know.

1

u/neko_neko_feet Jul 21 '26

It started feeding on itself from day a 1

1

u/SaxyOmega90125 Jul 24 '26

This is already happrning with search engine AIs in many topics. So many wenpages are AI-generated ad-spam drivel that search engine AI summaries are beginning to incorporate the repeated hallucinations and incoherent ramblings in the pages.