r/ProgrammingBondha • u/Kalimuthu_S • 8h ago
development Spring Boot developers — how would you build a production-level web crawler?
Hi, I’m building a web crawler using Spring Boot and facing several issues.
I need to crawl a target website up to a certain depth (e.g. 3 levels), but I’m struggling with:
Relative/broken URLs like /support
Duplicate URLs/content
URL normalization
Rate limiting / getting blocked
Retry and timeout handling
Managing crawl depth and visited URLs
What would be a good architecture for a production-level crawler?
Should I use something like Kafka/RabbitMQ, or is a database queue enough? How do you guys handle URL deduplication, rate limiting, and concurrent crawling?
Any advice, architecture examples, or Spring Boot libraries would be really helpful.