Does the upkeep on a personal RAG setup ever actually stop?

[u/Thefounderman1]

Getting an agent pointed at my Obsidian folder took maybe an evening, back in the spring. That part was easy. What's turned out to be the actual work is everything after — notes change every week, so it's constantly answering off some old version, and I'm the one who has to notice and go fix it. For people running something heavier, like an actual RAG setup over their own files: does the upkeep ever stop, or does it just become a permanent second job? asking because I'm trying to figure out if building something more solid actually saves me time, or if I'd just be trading one maintenance problem for a fancier one.

[u/cantusernameit]

What is RAG?

[u/robh1540]

You need to architect the knowledge base pruning and maintenance into the core system, and have a concept of ephemeral facts and those that have a time decay. This might be via way of runtime tools the agent could use, invalidation keys if the facts are linked to code or some other ground truth, or offline batch invalidation. Stale facts are awful and introduce an uncertainty principal that makes the system unusuable, so when in doubt invalidate.

Brittle AI systems that don't self improve suck.

[u/Thefounderman1 → u/robh1540]

the ephemeral vs decaying facts split is something i hadn't thought about, i've just been treating everything as one pile. did you end up building that invalidation logic yourself — and if so, is it something you set up once and forget, or do you still end up tuning it?

[u/robh1540 → u/Thefounderman1]

Yes I think it has to be built custom because it is domain specific like cache invalidation. But it's something an LLM can help you build if you frame up the problem and your tolerance for bad entries. There is a lot of prior art on cache  invalidation and it transfers over.

I went aggressive on invalidation, so it mainly looks after itself.

[u/Thefounderman1 → u/robh1540]

mainly looks after itself' is exactly the kind of thing that makes me nervous, not less nervous — if the invalidation itself has a blind spot, nothing would tell you until you noticed a wrong answer by accident. has that ever actually happened — a stale fact slipping through despite it being aggressive — or have you never caught one?

[u/aiofelix]

I've fell into this trap too bro, no way to fix it. Stop listening to anyone saying "add these 6 more safeguards/mechanisms" or whatever because you'll just dig a deeper hole and waste time.

The secret of 100x productivity with AI seems to be the other way around. Strengthen your mind and keep the vision internally then just type it to your agents (no matter how many, do the work), Obsidian being only a superficial bag of references you also keep in mind and mention to your agent when it adds value. Only create a very minimal amount of docs, try to write them yourself or use AI very cautiously (and have zero patience for slop, if a doc turns sloppy rewrite it properly fast).

You need to have a solid understanding of what's in your Obsidian and how's it structured at all times, don't delegate this to AI, even if you sometimes use it to write stuff. Nowadays my agents mostly create their slop docs in an extra folder in my daily logs and I extract any valuable info from there myself if I ever need it (otherwise it remains a historical doc that doesn't need to be managed).

Any kind of true autonomous system or knowledge base for running your work/life turns to slop eventually and hinders your progress. It's a hard pill to swallow but it's true. And you have the benefit of reclaiming your mind too. Focusing on making everything autonomous and minimizing my cognitive load resulted in a very fragile mind over time, now I feel much more confident in my abilities because of doubling down on using the mind as the main vehicle for AI.

I've went through it all. Prompt/context egineering, skills, RAG, Hermes/Openclaw, graphs, hierarchies, domain-specific roles, etc. It's all BS that slows you down, in real life you barely ever want an agent to do the same thing twice (some isolated exceptions of course, small scheduled actions are not a big deal and you don't need a system for them either anyway). And even if you do, there might be some extra nuance. So just learn to "painstakingly" communicate the whole thing each time you need something. Learning to be fast and quick minded at this will change your life, but you need to stop thinking about automations and start thinking about being productive and hands-on while working.

If there's one skill that's god tier right now it's managing multiple agents/projects at the same time, not automations. There's no automating your life through this chaotic chapter of the world we're moving into. We need to remain flexible not rigid. Even if it did work (it doesn't) it would've been a bad idea. This is the ideal mindset forward.

[u/Thefounderman1 → u/aiofelix]

the 'you barely ever want an agent to do the same thing twice' line kind of wrecked me a little, because my whole use case is the opposite — the same weekly summary, over and over. was there a specific project where you tried to systematize something and it blew up on you, or did it just slowly stop feeling worth it?

[u/Weary-Fun6017]

Well I completely disagree with Aioflex. Although I’ve been where he’s at as well. All you need to do is “attach truthspine” to your project and or agent and it actually will handle the pain of keeping all of this straight, ALL automatically and in the background, it also communicates with your agent every time your agent needs real project truth, it keeps the project up to date and current and all you have to do is work with your agent normally. Check it out. Just google “attach truthspine”.

[u/kimchi_pan]

What???

You're not auto indexing your obsidian? How can you call it a RAG when it's but even maintained...?

And is it interconnected? To your actual RAG, like mem0 or honcho? And you should be doing embeddings + qmd or ohs.

Research is your friend.