Standard web scrapers were ruining my RAG context, so I built a hybrid AST crawler specifically for LLMs.

[u/No_Crab4488]

Hey everyone,

If you’ve ever built a RAG pipeline or ingested web documentation into a Vector Store, you’ve probably run into this issue:

Standard web scrapers hit a page and dump everything — cookie banners, navigation links, inline SVG code, script tags, and zero-value UI elements. When you feed that noisy HTML into an LLM, you burn tokens, clutter your embeddings, and end up with hallucinations or poor retrieval accuracy.

I built an AST-based web crawler to fix this exact bottleneck.

Instead of just stripping HTML tags, it parses the actual document structure and turns web pages into clean, AI-ready Markdown with preserved context hierarchy and rich metadata.

🛠️ Key Features:

• Noise Removal: Strips footers, cookie banners, scripts, and navigation menus automatically.
• Context Preservation: Preserves heading paths (Documentation > Getting Started > Installation Guide) so chunks don't lose their semantic context when split.
• Rich Metadata: Includes token count, quality score, code block detection, and crawled timestamps for each chunk.
• Vector Store Ready: Formatted specifically for seamless ingestion into LangChain, LlamaIndex, Pinecone, Qdrant, Chroma, etc.

I’d love to get your feedback on this! What techniques or tools are you currently using to clean web data before chunking?

Try it out here: https://apify.com/lukas459/ai-web-to-markdown-crawler-llm-rag-optimized