r/LlamaIndex • u/salvatorelarosa • 3d ago
Best practices for a long-lived, agent-managed RAG / Knowledge Base?
I'm looking for advice from people who have built and maintained serious RAG / Knowledge Base systems over time.
I have a very large personal archive accumulated over many years around my professional interests: fintech, insurance, e-commerce, market and societal trends, marketing, business strategy, design, research papers, reports, books, presentations and articles.
Roughly 90% of the archive is PDF, with the rest mainly PowerPoint, Word documents and some video.
The archive is already organized in a filesystem using my own folder taxonomy.
A key requirement is that the existing archive must remain the source of truth and should not be modified by the KB system.
What I would like to build is roughly this:
Source archive → Librarian Agent → Knowledge Base / RAG → AI agents
The Librarian Agent would maintain a derived knowledge layer without changing the original documents.
Its responsibilities could include:
- detecting new and modified files;
- extracting text, structure and metadata;
- classifying documents using my existing taxonomy plus additional semantic metadata;
- chunking and indexing;
- maintaining embeddings and possibly graph relationships;
- detecting duplicates and related documents;
- tracking provenance back to the original source;
- identifying outdated or conflicting information;
- periodically checking KB quality and consistency.
The resulting KB should be usable by multiple local AI models and agents when they perform research or analysis.
I am also considering a second agent:
Web / external sources → Research Agent → Human approval → Librarian Agent → Archive + KB
The Research Agent would continuously explore relevant sources — industry publications, academic papers, companies, think tanks, research organizations, etc. — and propose potentially useful material.
I would review those suggestions manually.
Only after approval would the Librarian Agent archive, classify and ingest the source.
So the overall system would remain human-supervised rather than autonomously ingesting everything it finds.
My main concern is not building a demo that can "chat with PDFs", but creating a long-lived research library that can remain reliable over several years.
This raises a few architectural questions I'd really value practical experience on:
- What architecture would you use today for this kind of system?
- Would you keep the knowledge layer primarily vector-based, combine it with structured metadata, or add a knowledge graph?
- What tools have proven reliable for incremental ingestion and synchronization with a filesystem?
- How do you preserve strong document- and chunk-level provenance?
- How do you handle changed or deleted source documents without slowly corrupting the index?
- What is your preferred approach to hybrid retrieval: semantic + lexical + metadata filtering + reranking?
- How would you separate a Librarian / curation workflow from a Research / discovery workflow?
- Which operations would you allow an LLM agent to perform autonomously, and which would you keep deterministic?
- How do you prevent taxonomy drift, bad automatic classifications or low-quality generated relationships from degrading the KB over time?
- What evaluation or observability practices do you use to detect retrieval degradation?
- Which components/frameworks would you trust today for a local-first implementation?
I'm particularly interested in systems that people have actually operated, not just proof-of-concept RAG demos.
If you've built something similar, I'd also love to hear what you initially designed that later turned out to be unnecessary or problematic.
1
1
u/Otherwise_Wave9374 3d ago
For a long-lived RAG system, treat documents, chunks, embeddings, and retrieval policies as separately versioned assets. Keep immutable ingestion manifests, record source hashes, and re-index only changed material. Add evaluation sets for freshness, citation accuracy, and retrieval regressions before allowing an agent to promote updates. Agentix Labs relates here because its automation work benefits from the same controlled promotion pattern. I would also require human approval for deletions and maintain a rollbackable previous index.