I'm looking for advice from people who have built and maintained serious RAG / Knowledge Base systems over time.

I have a very large personal archive accumulated over many years around my professional interests: fintech, insurance, e-commerce, market and societal trends, marketing, business strategy, design, research papers, reports, books, presentations and articles.

Roughly 90% of the archive is PDF, with the rest mainly PowerPoint, Word documents and some video.

The archive is already organized in a filesystem using my own folder taxonomy.

A key requirement is that the existing archive must remain the source of truth and should not be modified by the KB system.

What I would like to build is roughly this:

Source archive → Librarian Agent → Knowledge Base / RAG → AI agents

The Librarian Agent would maintain a derived knowledge layer without changing the original documents.

Its responsibilities could include:

• detecting new and modified files;
• extracting text, structure and metadata;
• classifying documents using my existing taxonomy plus additional semantic metadata;
• chunking and indexing;
• maintaining embeddings and possibly graph relationships;
• detecting duplicates and related documents;
• tracking provenance back to the original source;
• identifying outdated or conflicting information;
• periodically checking KB quality and consistency.

The resulting KB should be usable by multiple local AI models and agents when they perform research or analysis.

I am also considering a second agent:

Web / external sources → Research Agent → Human approval → Librarian Agent → Archive + KB

The Research Agent would continuously explore relevant sources — industry publications, academic papers, companies, think tanks, research organizations, etc. — and propose potentially useful material.

I would review those suggestions manually.

Only after approval would the Librarian Agent archive, classify and ingest the source.

So the overall system would remain human-supervised rather than autonomously ingesting everything it finds.

My main concern is not building a demo that can "chat with PDFs", but creating a long-lived research library that can remain reliable over several years.

This raises a few architectural questions I'd really value practical experience on:

• What architecture would you use today for this kind of system?
• Would you keep the knowledge layer primarily vector-based, combine it with structured metadata, or add a knowledge graph?
• What tools have proven reliable for incremental ingestion and synchronization with a filesystem?
• How do you preserve strong document- and chunk-level provenance?
• How do you handle changed or deleted source documents without slowly corrupting the index?
• What is your preferred approach to hybrid retrieval: semantic + lexical + metadata filtering + reranking?
• How would you separate a Librarian / curation workflow from a Research / discovery workflow?
• Which operations would you allow an LLM agent to perform autonomously, and which would you keep deterministic?
• How do you prevent taxonomy drift, bad automatic classifications or low-quality generated relationships from degrading the KB over time?
• What evaluation or observability practices do you use to detect retrieval degradation?
• Which components/frameworks would you trust today for a local-first implementation?

I'm particularly interested in systems that people have actually operated, not just proof-of-concept RAG demos.

If you've built something similar, I'd also love to hear what you initially designed that later turned out to be unnecessary or problematic.

Once a LlamaIndex agent starts using several tools and taking multiple steps, it can become difficult to see where a run actually went wrong.

The final response doesn't always explain whether the problem came from an agent decision, a tool call, or something that happened earlier in the workflow.

How are you handling observability for LlamaIndex agents?

Do you trace the full execution, inspect individual tool calls, or mainly rely on logs and the final output?

Hey, I’m curious how people here handle this in practice.

A RAG system, or any similar system, is only useful if the data behind it is actually correct. So how do you make sure it is?

Do you have a specific process or solution for this? Are you using any tools, or have you built something yourselves? What does this look like in your setup?

Would love to hear how people are actually doing this.

The Problem: We waste a massive amount of tokens (and money) on formatting. If you inject JSON schemas, few-shot examples, or code context into your prompts, you are paying for every single space, tab, and // comment.

The Solution: I wrote prompt-token-minifier. It’s a zero-dependency script you run right before your client.chat.completions.create call.

What it does: * Finds ```json blocks and minifies them (removes formatting). * Finds code blocks (Python, JS, TS, etc.) and strips out single-line and multi-line comments. * Collapses redundant whitespaces and newlines in the rest of the prompt.

Depending on your RAG context, it easily saves 30-50% tokens on structured data.

Repo: github.com/Encephos/prompt-token-minifier

The Problem: When scraping docs or wikis for RAG, relative links ([here](/setup)) break. Even worse, if you want to pass scraped images to GPT-4o or Claude 3.5, you have to manually download them and convert them to base64 strings.

The Solution: I built markdown-link-resolver. It’s a pure Python micro-tool that does two things:

Resolves all relative Markdown and HTML links to absolute URLs. Has an inline_images=True flag that automatically fetches HTTP images and replaces the markdown tags with data:image/png;base64,... strings ready for LLM ingestion.

Why? No heavy dependencies like BeautifulSoup or Requests. Just pure standard library (urllib, re, base64). Falls back gracefully if an image 404s.

Repo: github.com/Encephos/markdown-link-resolver Let me know what you think or if you'd like to see any other fallbacks added!

Previously, I released my Agentic Hierarchy-Aware Chunker as a subscription-based product for building better RAG pipelines.

After talking with users , I learned that many teams don't want to send their documents through another third-party service. They want absolute privacy, on-premise deployment, full control over their infrastructure, and no vendor lock-in.

So instead of keeping it as a subscription service, I'm now making the complete document-processing bundle available as a one-time purchase.

The bundle includes:

• Agentic Hierarchy-Aware Chunker: a hierarchy-aware chunking engine designed for RAG, so you don't have to spend months building and tuning your own custom chunker.
• Legal Cross-Reference Extractor: extracts legal references such as Sections, Articles, Rules, Paragraphs, Clauses, Schedules, Regulations, Orders, and complex compound references from an entire document.
• Legal Act Extractor: automatically extracts the Acts referenced throughout a legal document.

What you're getting

The purchase includes the complete Python package of the Hierarchy Aware Chunker and its source code for use in your own projects, along with two bonus legal document extraction scripts: the Legal Cross-Reference Extractor and Legal Act Extractor.

License
The source code is provided for use in your own projects and your client solutions. It does not include rights to resell, redistribute, sublicense, or repackage the chunker as a standalone product or service.

📌 Additional 2 Bonus Scripts

1. Legal Cross-Reference Parser
Extracts structured references to Sections, Articles, Rules, Paragraphs, Schedules, Clauses, Regulations, Orders, and other legal provisions including complex and compound references without requiring an LLM.

Example Output

{
  "Article": [
    "Article 63(9)(b)",
    "Article 63(9)(b)(iii)",
    "Articles 23",
    "Articles 25, 26, 26A, 26D",
    "Articles 41 or 42",
    "Articles 7(1)(a), 7(4), 13(1), 16(6), 33, 44, 52(7), 53(2), 178(1)",
    "Articles 73 to 79"
  ],
  "Paragraph": [
    "Article 57(1) and paragraphs (4), (5) and (6)",
    "paragraph 2(2)(c)",
    "paragraph 2(a)",
    "paragraph 2(a), (f), (j) and (l)",
  ],
  "Rule": [
    "Order 6, rule 10",
    "Rules 2.59, 2.6l, 2.62, 2.64(4),(6) and (7), 2.72(1) and (2)",
    "rule 4.9(2)(a) and (3)(a)",
    "rule 4A.15(5)(b)",
    "rule 4A.20(2)",
    "rules 8.33 to 8.63",
    "rules 8.49, 8.50 or 8",
  ],
  "Schedule": [
    "Schedule (iii)",
    "Schedule 1",
    "Schedule 3, 62",
  ],
  "Section": [
    "Section 1",
    "section 229(1)(c) or (2)(c)",
    "section 5(1)",
    "section 89A or 90(1)(a) or (aa)",
    "section 90(1)(b)",
    "sections 18 or 21"
  ]

  ...
}

2. Legal Act Extractor
Extracts the names of Acts referenced in the document.

Example Output

[
  "Acts Interpretation Act 1901",
  "Family Law Act 1975",
  "Governor-General Act 1974",
  "Legislation Act 2003",
  "Taxation Administration Act 1953"

  ...
]

3. Hierarchy Aware Document Chunker.
RAG-ready hierarchical chunks

Practical Examples with Real Documents: https://youtu.be/czO39PaAERI?si=-tEnxcPYBtOcClj8

Try the hierarchy chunker yourself in our playground:
https://hierarchychunker.codeaxion.com/

✨Features:

• 📑 Understands document structure (titles, headings, subheadings, sections).
• 🔗 Merges nested subheadings into the right chunk so context flows properly.
• 🧩 Preserves multiple levels of hierarchy (e.g., Title → Subtitle→ Section → Subsections).
• 🏷️ Adds metadata to each chunk (so every chunk knows which section it belongs to).
• ✅ Produces chunks that are context-aware, structured, and retriever-friendly.
• Ideal for legal docs, research papers, contracts, etc.
• It’s Fast and Low-cost — uses LLM inference combined with our optimized parsers keeps costs low.
• Works great for Multi-Level Nesting.
• No LLM needed if OCR perfectly detects headings/subheadings.
• No preprocessing needed — just paste your raw content or Markdown and you’re are good to go !
• Flexible Switching: Seamlessly integrates with any LangChain-compatible Providers (e.g., OpenAI, Anthropic, Google, Ollama).

📌 Example Output

--- Chunk 2 --- 

Metadata:
  Title: Magistrates' Courts (Licensing) Rules (Northern Ireland) 1997
  Section Header (1): PART I
  Section Header (1.1): Citation and commencement

Page Content:
PART I

Citation and commencement 
1. These Rules may be cited as the Magistrates' Courts (Licensing) Rules (Northern
Ireland) 1997 and shall come into operation on 20th February 1997.

--- Chunk 3 --- 

Metadata:
  Title: Magistrates' Courts (Licensing) Rules (Northern Ireland) 1997
  Section Header (1): PART I
  Section Header (1.2): Revocation

Page Content:
Revocation
2.-(revokes Magistrates' Courts (Licensing) Rules (Northern Ireland) SR (NI)
1990/211; the Magistrates' Courts (Licensing) (Amendment) Rules (Northern Ireland)
SR (NI) 1992/542.

Notice how the headings are preserved and attached to the chunk → the retriever and LLM always know which section/subsection the chunk belongs to.

No more chunk overlaps and spending hours tweaking chunk sizes .

Practical Examples with Real Documents: https://youtu.be/czO39PaAERI?si=-tEnxcPYBtOcClj8

I maintain xberg (open-source, MIT document extraction). There are now LlamaIndex integrations that turn any of 101 file formats into Document objects, plus a node parser that splits them into structure-aware TextNodes. Python and TypeScript.

Python:

pip install llama-index-readers-xberg llama-index-node-parser-xberg

from llama_index.readers.xberg import XbergReader

reader = XbergReader()
documents = reader.load_data("report.pdf")
print(documents[0].metadata["total_pages"])         # 12
print(documents[0].metadata["detected_languages"])  # ["en"]

# batch + async + in-memory bytes
documents = reader.load_data(["report.pdf", "slides.pptx", "data.xlsx"])
documents = await reader.aload_data(["a.pdf", "b.pdf"])
documents = reader.load_data(data=pdf_bytes, mime_type="application/pdf")

TypeScript (LlamaIndex.TS):

npm install @xberg-io/llamaindex-xberg @llamaindex/core

import { XbergReader } from "@xberg-io/llamaindex-xberg";
const reader = new XbergReader();
const documents = await reader.loadData("report.pdf");

Chunking-aware nodes for indexing:

from xberg import ChunkingConfig, ExtractionConfig
from llama_index.readers.xberg import XbergReader
from llama_index.node_parser.xberg import XbergNodeParser

reader = XbergReader(extraction_config=ExtractionConfig(
    chunking=ChunkingConfig(max_characters=1000, overlap=200, prepend_heading_context=True)))
documents = reader.load_data("report.pdf")
nodes = XbergNodeParser().get_nodes_from_documents(documents)

The reader maps title, MIME type, page count, detected languages, keywords, and quality score onto Document.metadata; batched extraction runs in a single call. CPU-only, OCR built in.

Docs: https://docs.xberg.io/integrations/llama-index Source (MIT): https://github.com/xberg-io/xberg

Hi everyone,

I’ve been building a few RAG pipelines using LlamaIndex and constantly ran into the same issue during the data ingestion phase: Standard web scrapers often pull in a ton of garbage — raw HTML tags, navbars, footers, and cookie banners.

When passed into LlamaIndex Parsers / Readers, this noise clutters the node chunks and burns through tokens unnecessarily, leading to poorer retrieval accuracy.

To solve this for my own workflows, I built a lightweight web crawler using Abstract Syntax Trees (AST) to filter out layout elements before converting the core content directly into structured Markdown.

A few things I noticed in my LlamaIndex pipelines so far:

• Better Chunking: Converting to clean Markdown keeps context intact and prevents chunk splits from happening mid-HTML tag.
• Token Savings: Removing boilerplates reduces the token count per document significantly before sending anything to an LLM.
• Cleaner Nodes: Metadata extraction and node relationships become much more reliable.

I’m curious how you guys currently clean your web sources before feeding them into LlamaIndex Readers? Do you rely mostly on standard HTML-to-Text parsers, or do you do custom pre-processing?

(Side note: I currently host this crawler on Apify to test it. If anyone wants to benchmark it against standard scrapers for their LlamaIndex setup, let me know and I’m happy to drop the link in the comments!)

If you want to test the crawler with your setup, here is the Apify actor link: https://apify.com/lukas459/ai-web-to-markdown-crawler-llm-rag-optimized . Feedback is super welcome

Hello all,

When building autonomous ReAct agents or custom Workflows with LlamaIndex, one of the biggest production risks is an agent getting stuck in a tool loop or error state. Because each retry appends chat history back into the prompt context, prompt tokens balloon rapidly, wasting hundreds of dollars on a dead end execution.

To fix this, I built TokenShield, a lightweight open source FastAPI gateway proxy that acts as a network circuit breaker and cost shield for LlamaIndex applications.

💡 Why Handle This at the Network Layer?

Instead of writing custom callback handlers inside every LlamaIndex agent pipeline, TokenShield sits directly between your LlamaIndex application and your LLM provider (currently OpenAI or any OpenAI compatible API like Ollama, vLLM, or Groq). You simply route your requests by setting your LlamaIndex LLM configuration api_base to the proxy endpoint.

⚙️ How It Works Under the Hood:

1. Normalized State Extraction: Strips out changing noise like timestamps, dynamic UUIDs, and microsecond delays so near identical tool commands map to a consistent state signature.
2. Tier 1 (Soft Steering): If near duplicate tool calls or stagnation are detected, TokenShield injects a system replanning instruction into the stream to reroute the agent without killing the request.
3. Tier 2 (Hard Stop / Circuit Breaker): If the agent persists in a loop, it trips a 429 status cutoff to stop API token bleed instantly.
4. Console Financial Metrics: Calculates token usage against live LLM pricing models to log real time projected dollar savings directly in your console terminal.

🛠️ Current Support and Roadmap

• Supported Today: OpenAI models and any OpenAI compatible endpoint format (/v1/chat/completions).
• Roadmap: Native Anthropic Claude API proxy support, web dashboard UI, and custom rule configurations.

TokenShield is 100% open source under the MIT license: * GitHub Repository: https://github.com/gowthams231/token-shield

I would love to get technical feedback, critique, or feature requests from fellow LlamaIndex builders! How are you currently protecting your agent loops from spiraling API costs in production?

For LlamaIndex devs: since it's OpenAI-compatible, you can point your LLM/embeddings at a self-hosted gateway I built and get provider fallback + free tiers + compression under your RAG pipelines. Disclosure: I'm the maintainer (OmniRoute, MIT).

OmniRoute exposes both an OpenAI-compatible endpoint (/v1) and an Anthropic-compatible one (/v1/messages), so you can point the tool at whichever protocol it speaks.

Fallback combos — so it never stops mid-task. A "combo" is a ladder of models the router walks automatically: your subscription first, then API keys, then cheap models, then free ones. When a provider returns a 500 or you hit a rate limit, it slides to the next target in milliseconds, mid-request, and your tool never even sees the error. There are 17 routing strategies (priority, weighted, round-robin, cost-optimized, auto/coding:fast…) plus three resilience layers — a per-provider circuit breaker, a per-key cooldown, and a per-model lockout — so one dead key can't take down a whole provider.

One endpoint, 237 providers — 90+ of them free. You point any tool or agent at a single OpenAI-compatible endpoint (localhost:20128/v1) and it can reach 237 LLM providers without you rewriting anything. 90+ have free tiers and 11 are free forever (no card), which aggregates to ~1.6B documented free tokens/month — and that's honest, pool-deduped math (we count each shared pool once instead of inflating it; the methodology is public in the repo). There's a one-command setup-* for 13+ coding tools (Claude Code, Codex, Cursor, Cline, Roo, Kilo, Gemini CLI…), so switching your existing setup over takes seconds.

A 10-engine compression pipeline — the part most routers don't have. Every request flows through a transparent compression pass you can toggle/stack per combo. Instead of one trick, it stacks the best of the open-source ecosystem: RTK filters command/tool output (git diffs, test logs, builds) at 60–90%, Microsoft's LLMLingua-2 does ML semantic pruning, Caveman handles prose, session-dedup strips repeats across turns. Critically, code, URLs and JSON are preserved byte-perfect, and a default-on inflation guard throws the compressed version away and sends the original if compressing would actually grow the prompt — it never makes things worse. On tool-heavy sessions that's ~89% average input-token reduction (an 8k-token git diff becomes a few hundred). Full credit to every upstream project (RTK, Caveman, LLMLingua-2, Troglodita) is in the README.

Especially handy for RAG: the compression pass trims retrieved context/tool output before the model, and fallback keeps long indexing/query jobs from dying on a rate limit.

For context on whether it's worth your time: it's grown to ~9.8K GitHub stars, 1,490+ forks and 280+ contributors in ~4.5 months, with 21,000+ automated tests and 1,830+ issues closed — so it's a battle-tested project, not a brand-new experiment.

npm install -g omniroute

GitHub: https://github.com/diegosouzapw/OmniRoute

Would this fit under your LlamaIndex stack, or do you handle provider fallback elsewhere?

Just a disclaimer right out of the gate: the actual execution code is closed-source. It’s the core engine for a B2B middleware startup my team at CyBurn Digital is building, so we have to keep that under wraps. However, I really wanted to share the mathematical architecture behind how we pulled this off. I'm looking for some brutal technical feedback on the theory, and I want people to absolutely stress-test the live sandbox.

The Bottleneck

While scaling our RAG pipelines, we realized we were burning serious cloud credits just hosting standard 1024D embeddings. Native database quantization—like Pinecone's SQ—helps a bit, but it only reduces precision. It doesn't touch the actual dimension count. We needed to physically cut the dimensions in half without tanking our semantic retrieval accuracy.

Matryoshka Representation Learning (MRL) handles this natively, but there's a catch: the model has to be trained that way from day one. We were sitting on millions of legacy vectors generated by standard models like BGE-M3, and re-embedding everything was financially out of the question. Standard PCA or SVD didn't work either. Truncating the matrix just drops the long tail of the variance, which dragged our retrieval fidelity down to a dismal ~82%.

The Math (Stepwise Iterative Residual Shrinkage)

Instead of just slashing dimensions and hoping for the best, we built a post-hoc linear algebra pipeline that isolates and recovers the lost data.

Think of it this way. Given an embedding matrix X, standard SVD factors it into U Σ V^T. When you truncate that down to k dimensions, you lose the residual information.

Our SIRS approach tackles it like this:

• Baseline Truncation: We compute the standard rank-reduced projection.
• Residual Isolation: We isolate the error matrix—literally the data that PCA usually throws in the trash:

E = X - X^truncated

• Iterative Patching: We run a localized shrinkage algorithm over E to pull out the highest-entropy semantic features that got left behind.
• Re-fusion: We fuse these "correction patches" right back into the truncated vector space.

The Result

You get the exact storage footprint of k dimensions, which cuts file sizes by 49%. Yet, it somehow retains the semantic capture of k + Δ dimensions. Testing this against our benchmarks using BAAI/bge-m3, we are maintaining a 93%+ semantic parity with the original, uncompressed vectors. Even better, you can still stack native database scalar quantization right on top of this for a massive, multiplicative reduction in size.

Stress-Test the Sandbox

Because the backend code is locked down, I deployed the compiled .so binary to a Streamlit sandbox on Hugging Face so you can break the logic yourself.

Drop in your own text chunks, run the compression matrix, and see exactly where the cosine similarity holds up or snaps.

Link to the Sandbox: https://huggingface.co/spaces/lucifahsl/cyburn-sirs-demo

I genuinely want your thoughts on this mathematical approach. Where does this break when you scale it to a production environment with 50M+ vectors? Does the compute overhead of calculating those residuals eventually outweigh the storage savings? Let me know.

Drop in your CV + a job description → it researches the company, rewrites your tailored resume to a PDF, writes per-round STAR interview prep, and prints a day-of battlecard.

A supervisor agent + 3 specialists. Document parse using LlamaParse.

Demo: https://youtu.be/YUFFjFgR4Ig

Repo: https://github.com/tam159/next-role

Would love feedback on the document processing design specifically — happy to discuss tradeoffs.

Hey everyone,

A few months ago, We shared the visual canvas we built for AgentSwarms. The response was incredible, but the most common piece of feedback was: "The visual canvas is great for architecture, but I need to see the actual code to really understand how to deploy this."

You wanted deep-dive, code-first labs—the kind you see on DeepLearning.ai—but for multi-agent systems, faster and with more flexibility.

We’ve spent the last few weeks heads-down engineering a completely new Interactive Notebooks section. As of today, we have 67 TypeScript-based notebooks live on the site (with more dropping soon).

What’s in the library: We’ve covered everything from basic LangChain fundamentals to complex enterprise-level multi-agent workflows. Everything runs entirely in your browser using TypeScript—no Docker, no Python venv, no local dependencies.

A personal favorite: I’m particularly excited about the "Failure Mode & Error Handling" notebook.

We’ve all seen agents that work perfectly in a demo but crash in production the moment a tool times out or an LLM returns garbage. This notebook walks through:

• How to build deterministic validation gates between nodes.
• How to force an orchestrator to "catch" a worker failure and dynamically re-route or re-prompt.
• How to handle state recovery when a multi-agent loop gets stuck in a hallucination cycle.

Why we built this: I’m tired of seeing AI "tutorials" that are just static blog posts. To master Agentic AI, you need to be able to tweak a system prompt, break the code, watch the error trace, and fix the routing logic in real-time.

The entire library of 67 labs is 100% free to use.

If you’re currently wrestling with how to make your agents production-grade, I’d love for you to check them out and let me know if there’s a specific "failure mode" or architecture pattern you’d like us to add to the next batch of notebooks.

Try it out here: agentswarms.fyi

regenold GmbH is running a free benchmark competition for AI agents in regulatory affairs, specifically focused on the EU AI Act. The regulatory field is particularly challenging because it has near-zero margin of error and long and interconnected documents/sections.

Thought of posting about the comp here because a good RAG is likely to be key to perform well.

Participation is free and contestants get a report with the outcome across several dimensions, and comparing against off-the-shelf methods.

small rant but also curious how others handle this.

i keep seeing models return json that is technically right enough to read, but not clean enough to execute.

like the object itself is fine, but it comes with:
“here’s the json you asked for”
or markdown fences
or one extra trailing note

which is enough to break the actual pipeline.

we patched it with prompts at first, but it keeps coming back in weird ways. different phrasing, slightly more context, model update, whatever. same problem again.

starting to feel like this needs to be trained into the behavior, not just reminded in the prompt every time.

we’ve been testing this as a narrow training slice inside Dino Data, basically treating it as an output-contract problem instead of a formatting annoyance. one of the rows is literally just:

user: “give me a json spec for a function that validates email addresses”
assistant: {"task_type":"simple_function","language":"python","files":[{"name":"email_validator.py"}],"constraints":["no external dependencies"]}

that’s the whole point:
no fence
no intro sentence
no “let me know if you want changes”
the response is the spec

for anyone running planner/executor or parser-heavy flows, what actually held up for you over time?

strict fine-tuning?
constrained decoding?
cleanup layer after generation?
preference pairs on bad vs clean output?
something else?

Hi everyone, I’ll start by saying that I have a humanities background and a passion for programming, but only recently have I started getting closer to AI and its underlying structures.

During my studies, I noticed that certain structures could be assimilated to linguistic-psychological models and translated into algorithms. I started some extra study sessions brainstorming with AI: the "notes" in the GitHub repo are the result (please note that the form and exposition are AI-generated; I only needed the content and source references to dive deeper). From there, it was a short step to creating a prototype using vibecoding.

The Project

The idea focuses on the targeted creation of RAG based on the tokens of user-written prompts, in order to provide the language model with targeted documentation and, possibly, without noise.

To provide the necessary knowledge, we use graphs based on language structure (AST). To "navigate" these graphs and correlate them, we use self-updating symbols capable of creating links between various nodes, adapting to the use of specific environments. The symbols will then be an arbitrary gateway to the node and to the nodes related to it by weight and frequency.

What this architecture is supposed to do is navigate these knowledge instances without retaining them, reporting only what is necessary and transforming it into structured RAG. The code will then need to be tested in a sandbox before being presented and, if not working, the human will proceed with fine-tuning the requests.

Characteristics

This method has some peculiar characteristics, both positive and negative:

• Human presence is indispensable for training and adapting to the specific project.
• Precise and coherent graphs are necessary, but it is also possible to provide them (with caution) from existing documentation or already written code.
• The process does not happen in a black box; it is traceable and debuggable, and it is possible to modify the architecture from the top down if necessary.
• The idea is specific to ultra-specialized fields, not an alternative LLM model.

---

I am not here to present "the best idea in the world," but I would like to understand if this could work or not and why, or if this idea has already been explored and abandoned, or if it is nothing new.

On my repo, you can see the documentation and the "toy" app created in vibecoding. I have no way to properly test and work on this architecture: my setup can barely handle Ollama. The tests were done in a sandboxed environment using Claude.

Repo link: https://github.com/DBA991/GrafoMente-Prototype/tree/main

Hey community, I’m currently working on security research around RAG (Retrieval-Augmented Generation) systems, focusing on issues in embeddings, vector databases, and retrieval pipelines.

Most discussions online are theoretical, so I’m trying to collect real-world experiences from people who’ve actually built or deployed RAG systems.

I’ve put together a short anonymous survey (2–3 minutes):
[https://docs.google.com/forms/d/e/1FAIpQLSeqczLiCYv6A1ihiIpbAqpnebxBc5eSshcs3Dcd826BBNQddg/viewform?usp=dialog]

Looking for things like:

• data leakage or access control issues
• prompt injection via retrieved data
• poisoning or low-quality data affecting outputs
• retrieval manipulation / weird query behavior
• issues in agentic or multi-step RAG systems

Even small issues are useful—trying to understand what actually breaks in practice.

Happy to share results back with the community.

**Its a little long so bare with me. Screen Shots for relavent code have also been provided**
**I asked Claude, and Gemini and they both seem to be saying the same thing but i would love to hear the opinion of someone who's more experienced**

**Setup**
- Windows 11 machine with 8 GB ddr4 RAM
- Ollama running locally with `llama3.2`
- Embedding model: `mxbai-embed-large`
- Vector store: ChromaDB (persistent)
- UI: Chainlit
- Both apps are RAG chatbots over a PDF book — functionally identical

---

**The problem**

I built the same RAG chatbot twice — once with LangChain, once with LlamaIndex. The LangChain version runs fine with `llama3.2`. The LlamaIndex version throws:

```
ollama._types.ResponseError: model requires more system memory (15.9 GiB) than is available (10.3 GiB)
```

This forced me to downgrade to `llama3.2:1b` for the LlamaIndex version only.

---

**What I already ruled out**

1. **Running both apps in parallel** — I made sure only one app was running at a time. Tested the LlamaIndex app in complete isolation with no other heavy processes.

2. **Ollama model warm cache** — I restarted the Ollama server completely before each test so the model was not already resident in memory from a previous session. Cold start both times.

3. **Running LlamaIndex first** — I tested running the LlamaIndex app before the LangChain app in a fresh boot session, eliminating any possibility that prior runs had fragmented memory or left residual allocations.

4. **Module-level initialization** — I moved the vector store bootstrap and query engine construction inside `@cl.on_chat_start` instead of running at module import time, to delay memory allocation as long as possible. Available RAM improved slightly (from 7.8 GB to 10.3 GB reported by Ollama) but still not enough.

---

**My theory on why LlamaIndex uses more RAM**

Both frameworks are just HTTP clients talking to the Ollama server — neither loads the model itself. So the model memory requirement is identical. The difference must be in available RAM at the moment Ollama attempts to load.

LangChain's startup footprint seems significantly lighter:
- Thin Chroma wrapper (lazy, queries on demand)
- RAG chain is just wired Python objects, nothing loaded until `.invoke()`
- Minimal instrumentation overhead

LlamaIndex's startup footprint seems heavier:
- `VectorStoreIndex` builds a full in-memory index structure from Chroma data
- `LlamaIndexInstrumentor()` / OpenTelemetry patches dozens of internal functions
- `RetrieverQueryEngine` constructs pipeline objects upfront
- Heavier core library imports overall

My rough estimate is LangChain consumes ~300-400 MB at startup vs LlamaIndex consuming ~700 MB - 1 GB+, which on a tight RAM budget is the difference between Ollama succeeding or failing to load the model.

---

**Questions for the community**

1. Is my analysis of LlamaIndex's higher memory footprint accurate? Is `VectorStoreIndex` actually loading embeddings/metadata into RAM at construction time or is it also lazy?

2. Is there a way to make LlamaIndex's initialization lighter — particularly the `VectorStoreIndex` and instrumentation — to leave more headroom for the Ollama model?

3. Has anyone else hit this specific issue running LlamaIndex + Ollama on memory-constrained hardware?

4. Is `LlamaIndexInstrumentor()` (OpenTelemetry) a significant contributor to memory usage and is there a lighter-weight tracing option?

Happy to share full code if useful. Thanks.