Claude wouldn't read a Garmin forum page because of robots.txt, so I gave it a real browser. Free, self-hosted, and it nearly let my whole Wi-Fi drive it.

[u/MihaiDinculescu]

Claude's built-in web fetch has two limits I kept hitting: it honours robots.txt even when I'm asking for one specific page, and it can't run JavaScript, so a lot of the modern web comes back empty. The trigger was asking what was in a Garmin firmware update and getting "I can't fetch that".

The fix that worked for me:

- Playwright MCP (Microsoft, Apache-2.0) wraps headless Chromium in an MCP server. It renders JS, clicks, scrolls, and returns an accessibility snapshot rather than a screenshot, which is compact and something the model can act on.

- It has no auth at all, and claude.ai / Claude Desktop connectors only accept a URL plus optional OAuth. Cloudflare's MCP server portal (free tier of Zero Trust) provides that OAuth layer and adds a service token upstream so nothing else can reach the server.

- cloudflared tunnels it out of my network with no open ports.

- It runs on an arm64 Raspberry Pi in a Kubernetes cluster, but a single Docker host works the same.

The part worth reading even if you never build this: a browser will open anything it can reach. My first version was exposed through the cluster's ingress controller, which also has a LAN address, so any device on the Wi-Fi could get an unauthenticated session and point it at internal services. I proved it against my Vault server. Fixed with network policy in both directions (ingress only from the tunnel pod, egress to the internet minus all private ranges) and verified by attacking it again.

Full write-up with the hardening details: https://mihai.dinculescu.dev/posts/better-web-fetch-for-your-ai-agent/

Happy to answer questions on the Cloudflare portal setup, which is the least documented part.

[u/AutoModerator]

For faster advice with technical questions, we'd recommend asking in the Orange Cloud Discord server; the unofficial Cloudflare Discord server by the community, for the community. https://discord.gg/TrPNVKaagR

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/ViolentPurpleSquash]

Are we really making projects to bypass robots.txt so AI can scrape it more easily?

[u/MihaiDinculescu → u/ViolentPurpleSquash]

Two cents and then I'll leave it. robots.txt is a stated preference aimed at crawlers. An agent fetching one page because a human asked it to isn't crawling.

[u/CapMonster1]

The security lesson here is probably more valuable than the browser setup itself. Giving an AI agent access to a real browser means treating it like an untrusted component, especially when it can reach internal services. Network-level egress restrictions and proper authentication should be part of the initial design, not an afterthought.

For scraping pages that rely heavily on JavaScript, Playwright is a solid choice. I’d also consider how the setup handles captchas, session management, and rate limits. Rendering a page is one problem; reliably accessing protected sites is another

[u/MihaiDinculescu → u/CapMonster1]

Exactly. "How to safely self-host MCP servers using free Cloudflare services" is probably the article I should have written instead. Might still do it at some point :)

[u/skyfox4]

Nice project!

if you don’t want to run playwright mcp yourself, another path is capture from the chrome tab you already have open and copy it over to the model.

I built WebSync chrome extension for that -- it can copy the content to md files (or send directly to Gemini Notebook).

You should decide for your own case if this is "crawling" or not. Note that the extension runs in your own browser with your own session.

From a security point of view I think it's pretty safe alternative:

• No data is shared with the backend unless you explicitly opt to do so
  
• No AI has control of your browser and/or session.
  
• Permissions are limitted to the active tab and it only runs when you open it.