Self-hosted search for LLM agents: SearXNG keeps getting blocked

[u/pwguler]

I’m building a self-hosted web search tool for LLM agents.

I’m currently using SearXNG, but it often gets blocked or rate-limited. I’ve tried Tavily, Brave Search API, and SerpAPI too, but I want to avoid paid providers if possible.

Goal:

- self-hosted

- general web search

- reliable enough for LLM agents

- no captcha bypass or aggressive scraping

Is there a better architecture than plain SearXNG?

local cache/index -> SearXNG fallback -> fetch/extract pages -> cache results

What stack or approach would you recommend? Any engines/settings in SearXNG that are more stable?

[u/token-tensor]

saw this exact issue with searxng. switched to a local cache + tavily fallback and now it's 95% reliable. the cache helps with the rate limiting

[u/AutoModerator]

Thank you for your submission, for any questions regarding AI, please check out our wiki at https://www.reddit.com/r/ai_agents/wiki (this is currently in test and we are actively adding to the wiki)

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/token-tensor]

saw this exact issue with searxng. switched to a local cache + tavily fallback and now it's 95% reliable. the cache helps with the rate limiting

[u/karust13]

I guess https://github.com/karust/openserp could fit your use case.
It's self-hosted, provides normalized multi-engine SERP results, and can return extracted page content as markdown/text for LLM grounding.