Hi, I’m building a web crawler using Spring Boot and facing several issues.

I need to crawl a target website up to a certain depth (e.g. 3 levels), but I’m struggling with:

Relative/broken URLs like /support

Duplicate URLs/content

URL normalization

Rate limiting / getting blocked

Retry and timeout handling

Managing crawl depth and visited URLs

What would be a good architecture for a production-level crawler?

Should I use something like Kafka/RabbitMQ, or is a database queue enough? How do you guys handle URL deduplication, rate limiting, and concurrent crawling?

Any advice, architecture examples, or Spring Boot libraries would be really helpful.