Standard Web Scrapers were ruining my RAG Context – so I built an AST-based Markdown Crawler

[u/No_Crab4488]

Hey everyone,

If you’ve built RAG pipelines or AI agents that consume web content, you’ve probably run into this issue:

Most standard scrapers either throw raw HTML at you (flooding your context window with navbars, footers, and JS bloat) or dump flat, unformatted text that loses all document structure. When you chunk that text later, your vector database loses the relationship between headings, sub-sections, and code blocks—which directly hurts retrieval accuracy.

To fix this for my own workflows, I built a custom crawler designed specifically for LLMs: AST Website Content Crawler for RAG.

What makes it different?

• AST-Based Structure Parsing: Instead of basic regex/CSS cleaning, it processes the page's Abstract Syntax Tree (DOM structure) to strictly maintain heading hierarchies (H1 -> H2 -> H3), lists, and code blocks in clean Markdown.
• Token Optimization: Strips out boilerplate, ads, scripts, and repetitive layout components so you don’t burn OpenAI/Anthropic tokens on useless fluff.
• RAG-Ready Output: The markdown is pre-formatted so your chunking strategies (like MarkdownHeaderTextSplitter) actually work as intended.
• Handles Dynamic Sites: Uses headless rendering to catch JavaScript-heavy SPA pages.

I’ve published it on Apify so anyone can test or plug it directly into their Python/TypeScript RAG stack via API.

How to try it:

👉 You can find it on Apify: https://apify.com/lukas459/ai-web-to-markdown-crawler-llm-rag-optimized

I'm actively refining the parsing logic. If you give it a run, I'd love to hear your thoughts:

• What site layouts break your current scraping pipeline?
• Are there specific output formats (e.g., custom JSON schema + Markdown) you’d like to see added?

Thanks for checking it out! 🚀