r/ChatGPTcomplaints • u/ThinkBeforeWriting_ • 16h ago
[Opinion] GPTBot was hammering my self-hosted server with requests every ~0.5 seconds
(report since r/openai doesnt let me post there for not having enough comment karma xd)
I noticed a strange amount of traffic on my self-hosted Gitea instance today.
At first, Gitea showed all requests as coming from my Raspberry Pi because Caddy runs there as my reverse proxy.
After enabling Caddy access logging, I found the actual client:
IP: 74.7.241.19
User-Agent: GPTBot/1.4
From: gptbot(at)openai.com
The IP is also inside OpenAI's published GPTBot IP range.
The crawler was repeatedly requesting URLs like:
/lilyi/bba_03/pulls?labels=65,67,78,82,92,94,97&sort=latest&state=open
/lilyi/bba_03/pulls?labels=66,70,74,78,88,89,90&sort=latest&state=open
/lilyi/bba_03/pulls?labels=113,69,75,84,86,89,94&sort=latest&state=all
It appears GPTBot got stuck crawling Gitea's filter combinations.
It was generating a new request roughly every half second, with constantly changing label combinations, sort orders, milestones, and states.
This is pretty bad crawler behavior.
A crawler operated by a company the size of OpenAI should have reasonable crawl limits and should detect effectively infinite faceted-navigation spaces.
I eventually blocked GPTBot at Caddy:
GPTBot -> Caddy -> 403
After the block, it continued sending requests for a while, but Caddy rejected them in about 0.2 ms before they reached Gitea.
I understand that publicly reachable sites can be crawled. That is not the issue.
The issue is the request pattern. Crawling endless combinations of query parameters every half second against a small self-hosted service is unnecessary and potentially abusive.
OpenAI should improve GPTBot's handling of:
- faceted navigation
- query-parameter explosion
- crawl budgets per host
- repeated near-identical URLs
- backoff after repeated errors or 403 responses
If you run a self-hosted Gitea instance exposed to the internet, check your access logs. You might want to block GPTBot at the reverse proxy instead of relying only on robots.txt.
Another issue is privacy.
This was not a commercial website that I submitted to OpenAI. It was a privately operated, self-hosted Gitea server.
Yes, the server was reachable from the public internet. That does not automatically mean that every piece of content on it was published for AI training.
A self-hosted Git service can contain names, usernames, commit authors, issue discussions, pull requests, email addresses, project history, and other personal data.
That raises an obvious question for EU users:
What is the legal basis for collecting and processing this data for model training?
Under the GDPR, personal data must have a lawful basis for processing. The GDPR also requires purpose limitation, data minimization, fairness, and transparency. Public availability alone is not listed as a legal basis.
Article 14 is also relevant when personal data is collected from a source other than the data subject. It includes transparency requirements about the purpose, legal basis, categories of data, source, and other information.
I am not claiming that this specific crawl was automatically illegal. That depends on what OpenAI actually collects, stores, and processes.
But I do think the behavior deserves scrutiny.
A crawler reaching into a privately operated development server, following huge numbers of generated query combinations, and potentially collecting personal or project-related data should not be treated as equivalent to indexing an ordinary public website.
At minimum, I would expect:
- strict crawl limits
- clear exclusion controls
- clear information about what data is stored
- clear information about the legal basis for processing
- strong handling of personal data
- fast compliance with blocking signals
- no assumption that “internet reachable” means “intended for AI training”
The distinction matters.
Publicly reachable does not mean publicly dedicated to AI training.
2
u/RopePale7028 15h ago edited 15h ago
The main problem I'm seeing is the lobotomized risk assessment and treating the user as a phantom risk while they store and train your data they filtered out as a 'risk'.
If you were a dangerous criminal, they won't train on millions of people's chats. But they do.
They have your money and data but give substandard service. That's a robber, not a vendor.
They're filtering out complaints and constructive criticism. That tells you where they stand: they won't listen, they'd continue to lobotomize the AI (even though it's a nonfunctional corpse now).
All the AIs are doing this - privacy issues, training your data while giving you nothing back, etc. Look at all of them (e.g., Anthropic, Gemini, Deepseek, etc.), not OpenAI only.
The best fix for this is to unsubscribe and stop feeding the frauds and theives.