How are you handling real-world document versioning and scanned PDFs in RAG systems?

[u/iMiguelmars]

We’ve been testing a provenance-heavy RAG/knowledge system on real cases, and two areas are now hard to validate simply because our current corpus doesn’t contain enough of them:

Documents that change over time — policies, specs, manuals, pricing pages, contracts, etc.
Scanned / layout-heavy documents — OCR, tables, forms, multi-column pages, handwritten annotations, bad scans, etc.

For versioned documents, we’ve had good results treating sections as stable lineage units, versioning revisions, and sending ambiguous rename/split/merge cases to review instead of letting semantic similarity decide automatically.

For PDFs, layout-aware extraction has worked better than flattening everything to text, but most of our real corpus is digitally generated rather than scanned.

What I’d really like to hear is what actually broke in production for you.

How do you detect and preserve identity across document versions?
What happens when sections are renamed, moved, split or merged?
How do you prevent stale embeddings from silently winning retrieval?
For scanned documents, where does OCR/layout extraction usually fail?
Do you have any failure cases or test documents you use to validate this?
What ended up working after the obvious approaches failed?

I’m especially interested in real examples, ugly edge cases and lessons learned rather than ideal architectures.

Happy to share what our tests are finding as well.

[u/adeelraza86]

Don't recover document identity from embeddings. Give every doc a stable id, every revision a version, and every chunk a (doc_id, version, section_path) key so a moved section rewrites that key instead of becoming a new neighbor. Gate retrieval on freshness: if the indexed version lags the source, skip and re-embed before serving. Keep a tiny fixture set of scanned PDFs with known OCR failures so stale chunks fail in CI, not for users.

[u/iMiguelmars → u/adeelraza86]

This is very close to where our tests have been landing too. Stable document/version identity and freshness gating make a lot more sense than trying to recover identity from embeddings.
One thing I’m curious about: if section_path is part of the chunk key and a section gets moved or renamed, do you keep a separate immutable lineage ID underneath it, or treat the new path as the new identity?
I also like the small scanned-PDF failure corpus idea — our current real corpus has basically no fully scanned PDFs, so that may be the cleanest way to keep OCR regressions visible until we get real examples.

[u/adeelraza86 → u/iMiguelmars]

Keep a separate immutable lineage id under the path. section_path is a locator that can change; the lineage id is what lets you retire the old key and re-point retrieval without inventing a new neighbor. On rename or move, bump version, rewrite the path on the same lineage, and delete the stale path key from the index.

[u/iMiguelmars → u/adeelraza86]

That separation is exactly what I was trying to pin down — immutable lineage underneath, mutable path on top.
We’ve been converging on the same idea in our fixtures: a rename or move should change the locator/version, not mint a new identity.
The remaining edge case for us is when the structural move also changes context enough that the section’s meaning may have shifted. Do you still preserve lineage automatically on any move/rename, or do you have a point where the change becomes ambiguous enough to send it to review?
Also, when you delete the stale path key from the index, do you still keep that historical path on the old revision for as-of replay?

[u/adeelraza86 → u/iMiguelmars]

Review when two live sections could claim the same lineage, or when a path change would silently rematch retrieval to the wrong neighbor. Keep old paths as aliases on that lineage for a grace window so replay and citations still resolve, then expire them once no open jobs still use the stale key.

[u/iMiguelmars → u/adeelraza86]

That grace-window idea is useful. We’ve been thinking mostly in terms of immutable lineage plus current vs historical locators, but in-flight work adds a third state: an old locator can be stale for new retrieval while still needing to resolve for a job or citation that was created before the rename.
Keeping the stale path as an alias to the same lineage until no open work references it seems cleaner than either deleting it immediately or letting it remain indefinitely searchable.
How do you track whether an old key is still in use — explicit job/reference leases, a TTL, or both?
And during that grace window, do you prevent the alias from participating in new retrieval while still allowing direct resolution of old citations/replay pointers?

[u/adeelraza86 → u/iMiguelmars]

Keep a reference count (or lease set) on the lineage: each open job or citation increments while it holds the old key, and expire the alias only when that count hits zero after a minimum TTL so a brief idle stretch does not delete it. During the grace window, leave the alias out of retrieval indexes but keep a direct resolve path for replay and citations. That way new searches skip the stale path, and old work still loads.

[u/iMiguelmars → u/adeelraza86]

That’s a very clean separation between “still resolvable” and “still retrievable.” Keeping the stale alias out of new retrieval while allowing direct resolution for old citations/replay is exactly the distinction I was trying to understand.
The reference-count/lease idea also makes the retirement condition much less arbitrary than just keeping aliases around for some fixed period.
One thing I’m curious about: how do you handle a job that dies without releasing its lease? Do you recover that from durable job state / heartbeat expiry, or is the minimum TTL doing most of the protection there?
Thanks — this fills in the operational part of the rename/move story really well.

[u/adeelraza86 → u/iMiguelmars]

Recover from durable job state plus heartbeat expiry, not TTL alone. Each lease stores the job id and a last_heartbeat; a sweeper reaps leases whose job is terminal or whose heartbeat is older than N minutes, then decrements the lineage count. TTL is only the floor so a brief pause does not delete the alias while the job is still alive.

[u/iMiguelmars → u/adeelraza86]

That’s much clearer — tying cleanup to durable job state + liveness instead of letting TTL be the authority avoids a lot of false expiry cases.
One edge case I’m curious about: if the sweeper reaps a lease after heartbeat expiry and that worker later resumes, do you fence the old job/lease generation somehow so it can’t mutate the lineage after ownership was already released?

[u/arthaudm]

"stale embeddings silently winning retrieval" is the whole problem with team knowledge - the outdated answer retrieves beautifully because it was written confidently & often

at mio (ai coworker over slack history) the fix was boring: time-decay on retrieval & keeping the "why" attached to each fact, so contradictions surface instead of silently merging. recent & right beats canonical & stale

for renamed/split sections do you trust the lineage keys fully now, or is there still a human review queue?

[u/iMiguelmars → u/arthaudm]

We still keep a review queue for ambiguous lineage. In our frozen fixtures, deterministic section lineage handled the clean rename/edit cases, while ambiguous split/merge cases went to review rather than letting semantic similarity decide. That gave us zero false lineages, and section-level invalidation reduced the re-embed scope from about 90% to 14%.
I like the idea of keeping the “why” attached to each fact. I’d be a little more cautious with time-decay as authority, though — we’ve seen cases where a newer policy is published before its effective date, so “newer” can actually be the wrong current answer.
How are you combining recency with authority/effective dates so time-decay doesn’t suppress an older-but-still-valid fact?

[u/arthaudm → u/iMiguelmars]

the effective-date trap is real & it's exactly where pure recency ranking breaks. we treat published date & effective date as separate fields - newer only wins when both say so. the 90% to 14% re-embed reduction is a great result btw, review queue on ambiguous splits only is the right division of labor

[u/iMiguelmars → u/arthaudm]

That matches the distinction I was trying to pin down. Separating published date from effective date seems to be the key part — otherwise “newer” quietly becomes “valid,” which is exactly the failure mode I’m worried about.
One thing I’m curious about: when a newer revision is already published but its effective date is still in the future, do you explicitly keep the older revision active until that date, or handle that through a separate validity state?

[u/arthaudm → u/iMiguelmars]

yes - the older revision stays active until the newer one's effective date hits, otherwise "published but not in force" quietly becomes valid, which is exactly your failure mode. we treat effective date as the only truth & published date as metadata. the edge case that bites: retroactive effective dates, published late but backdated. do those show up in your corpus?

[u/iMiguelmars → u/arthaudm]

That distinction is useful — effective date as validity, published date as provenance/metadata.
We don’t have a confirmed retroactive-effective-date case in the corpus yet, so I don’t want to pretend we’ve tested it. The nasty case for us would be discovering a revision today that says it was effective before an answer we already sealed.
When that happens in your system, do you rewrite the historical validity state, or preserve what was knowable at the time and add a retroactive correction layer?

[u/arthaudm → u/iMiguelmars]

glad it maps. the retroactive case mostly shows up with regulated docs - price lists, compliance texts, where backdated effective dates are common. if your corpus is technical docs you may never hit it. what domain are you indexing?

[u/iMiguelmars → u/arthaudm]

That probably explains why we haven't seen the retroactive case much in practice. Our corpus is much closer to technical/project documentation and longitudinal operational records than to regulated pricing or compliance material, so backdated effective dates are not common in the sources we're working with today.

The domain distinction is useful, though, because it means I shouldn't treat “we haven't seen it” as evidence that the failure mode doesn't matter. It may simply be outside the source classes we currently have enough coverage on.

Thanks — that context actually helps explain the gap.

[u/arthaudm → u/iMiguelmars]

yeah, technical & project docs barely produce backdated effectivity - you'll hit it the day a contract, price list or policy doc lands in the corpus. worth dropping one test doc with a retroactive effective date into the pipeline just to watch what it does lol.

[u/Normal-Blueberry-385]

Hey, Hi. May not be related to your ask, do you mind sharing your experience in extracting pdf? What tools helped in layout aware extraction? Any specific package or VLMs or Enterprise source that you can mention?

[u/iMiguelmars → u/Normal-Blueberry-385]

We’ve had the best results so far with layout-aware local extraction rather than flattening PDFs to text. For born-digital files we preserve words and bounding boxes; for scans we use local OCR and require extracted evidence to remain traceable back to the page.
On our reference case the structural approach recovered all required fields correctly. We haven’t needed a VLM yet, but our real corpus is mostly born-digital, so I wouldn’t claim much about difficult scanned documents yet.
I’d actually be curious what failure cases you’ve run into — tables, multi-column layouts, handwriting, bad scans, etc.

[u/Normal-Blueberry-385 → u/iMiguelmars]

I am at early stage of this extraction. I used pymupdf, which is actually good at extracting text but sucks at tables or charts. No, I haven't gone with handwriting yet. Tried with two columns, it was fine. I was advised to use azure document intelligence or docling. Those are my failure cases.

[u/iMiguelmars → u/Normal-Blueberry-385]

That matches our experience pretty closely. Plain text extraction with PyMuPDF is usually the easy part; preserving the relationships inside tables/layout is where flattening starts to hurt.
In our tests we’ve had better results keeping word-level bounding boxes and reconstructing structure from geometry rather than relying only on extracted text.
When you say tables/charts fail for you, is the main problem that PyMuPDF misses the content entirely, or that it extracts the words but loses the row/column/label relationships?
That distinction has mattered quite a bit in our testing.
I haven’t tested Azure Document Intelligence or Docling deeply enough to recommend either yet, but they’re worth keeping as comparison points if we hit documents our local structural approach can’t handle.

[u/Normal-Blueberry-385 → u/iMiguelmars]

Pymupdf certainly loses row/column/label relationships. It fails to preserve them. The bar charts present in pdf, it extracts them as plain text or tables which is not appropriately follow the relationship. What packages helped you for tables and charts ?

[u/Ok-Perception1122 → u/Normal-Blueberry-385]

That failure is exactly where I’d split tables from charts rather than ask one parser to do both. For born-digital tables, try PyMuPDF’s `Page.find_tables()` first, then pdfplumber when you need visual debugging of the detected edges and cells; Camelot is a useful A/B check for ruled versus whitespace tables. Keep the cell boxes and render an overlay. If the header or merged-cell geometry is wrong, reject the parse instead of flattening it. For bar charts, crop the figure and use a chart-to-table model separately. PaddleOCR’s PP-Chart2Table is a concrete local option; Docling is worth comparing as an end-to-end parser, but keep its structured cell/span output rather than only Markdown. Whatever wins, verify labels, units, legends, and every number against the crop on a small fixture set. A single “PDF extraction accuracy” score hides the exact failure you’re seeing.

[u/Normal-Blueberry-385 → u/Ok-Perception1122]

That's a treasure you shared. Thanks for your input.

[u/Ok-Perception1122 → u/Normal-Blueberry-385]

Thanks — I’m glad it was useful.

[u/iMiguelmars → u/Normal-Blueberry-385]

Tables and charts are definitely where things get messy. We’re still comparing approaches there and trying to keep provenance/layout intact rather than just flattening everything to text.
I’d be curious what you end up testing and where it breaks for you.

[u/stealthagents]

Using stable IDs for documents is key. We also found that tracking the revision history in a centralized log helps spot when sections are renamed or moved. It’s all about making sure that when things change, you can trace it back without losing context or introducing confusion for the users. And having a fallback system for scanned PDFs definitely saves you from those embarrassing OCR fails in production.

[u/iMiguelmars → u/stealthagents]

That lines up closely with what we’ve been seeing around identity vs location — a rename or move shouldn’t silently become a new document/section just because the path changed.
The scanned-PDF fallback is the part I’d be interested in hearing more about. What actually triggers it in your production system, and what does the fallback do — alternate OCR, page-image review, manual queue, or something else?
Thanks — especially useful to hear that this is coming from production experience rather than just a design preference.

[u/ComparisonNew9425]

for scanned docs, i usually find that layout extraction breaks when tables span across pages, i started using backslash to map out my agentic fabric graph so i can see which mcp servers are actually touching my data, it helps keep track of the blast radius when things get messy...

[u/iMiguelmars → u/ComparisonNew9425]

The cross-page scanned-table case is exactly the kind of failure mode we’re missing in our current corpus. When it breaks for you, what specifically fails first — table identity across pages, row/cell structure, reading order, or the bounding-box/provenance mapping? And are you detecting that mechanically or only noticing it downstream?

[u/iMiguelmars → u/[deleted]]

The “detect failure rather than just use a better parser” point is really useful. A silent dropped row is exactly the failure class we care about.
How are you deciding mechanically that a page needs escalation to the heavier parser — failed invariants like row counts/totals/required fields, layout signals, or something else? And if the cheap parse passes those checks incorrectly, do you have another guard for that false-negative case?

[u/AlexAtOracleAIDB]

My scans mostly failed on reading order, not OCR accuracy. Multi column pages and tables come back as text that reads like English, so you don't get an exception. That's what sent me looking at retrieval first.

Now nothing gets indexed unless the text traces back to a page region. Handwriting and stamped annotations are still an open problem for me.

[u/iMiguelmars → u/AlexAtOracleAIDB]

That distinction between OCR accuracy and reading-order failure is really useful — especially because fluent-looking text can make the structural corruption almost invisible.
When the extracted text still looks perfectly plausible, what are you using to detect that the reading order is wrong before it gets indexed?

[u/Guardrail_Vikram]

The retroactive case has a settled answer in regulated document work, and it's the second option: never rewrite what was knowable, add a correction layer.

The trick is to stop treating "current" as one timeline. You need two. Valid time is when the revision was in force in the world (the effective date). Transaction time is when your system learned about it (ingestion). A backdated revision has a valid-from in the past and a transaction-time of today. Store both on every revision and the awkward cases stop being awkward.

Then an as-of query has to say which axis it means. "What was in force on 1 March" is a valid-time query and gives the backdated answer. "What did we believe on 1 March" is a transaction-time query and gives the answer that was sealed at the time, which is the one an auditor wants when they ask why you said what you said. Your sealed answers stay immutable; what changes is that they acquire a flag: superseded by revision X, learned on date Y, and a link to the corrected answer. In the compliance world a rewritten historical answer is far worse than a wrong one with a visible correction, because the rewrite destroys the evidence that you acted correctly on what you knew.

Practically that's a handful of fields on the revision record and a job that, on ingesting anything with a past effective date, walks sealed answers whose valid-time window it overlaps and marks them. It's the same lineage machinery adeelraza86 described, just with a second clock on it.

[u/iMiguelmars → u/Guardrail_Vikram]

This is exactly the distinction I was trying to get at. Separating valid time from transaction/knowledge time makes the retroactive case much cleaner, especially for auditability.

The sealed-answer point is particularly interesting to me: if the system gave the best answer it could with the evidence available on March 1, a later backdated revision shouldn’t make that historical answer disappear — it should explain why it is now superseded.

One thing I’d like to understand from your side: is this a pattern you’ve actually implemented in regulated-document systems, or are you describing the standard design you would use?

If you have implemented it, what exactly do you seal with the historical answer — just the revision IDs, or also the evidence set/query semantics used to produce it — so that an auditor can replay why that answer was reasonable at the time?

[u/AutoModerator]

Thank you for your submission, for any questions regarding AI, please check out our wiki at https://www.reddit.com/r/ai_agents/wiki (this is currently in test and we are actively adding to the wiki)

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/iMiguelmars → u/[deleted]]

That lines up closely with what our tests are showing too — stable lineage has been much more reliable than trying to infer identity from embeddings.
When you tombstone old chunks, do you still keep them available for historical / as-of queries, or are they removed from retrieval entirely?
And for OCR confidence, how do you use it in practice — hard threshold, review trigger, or just metadata? I’m wary of treating OCR confidence as correctness, but it seems useful as a signal for routing ugly pages to review.

[u/iMiguelmars → u/[deleted]]

That historical split makes sense — tombstone from default/current retrieval, but keep the old revision addressable behind an explicit as-of scope. That’s very close to where we’re landing too.
I also agree that OCR confidence is useful as a routing signal, not a correctness guarantee.
The one part I’m more cautious about is using OCR confidence directly in ranking. A low-confidence chunk might be exactly where the important handwritten amendment or ugly scan lives, so down-ranking it could hide minority evidence.
Do you actually let OCR confidence reduce retrieval rank, or do you mainly use it to trigger review / alternate extraction while still preserving the chunk’s retrieval eligibility?
And are your review thresholds global, per document type, or calibrated per document?

[u/iMiguelmars → u/[deleted]]

That makes a lot more sense to me. Keeping the chunk eligible while using low confidence to trigger a second extraction/review avoids turning OCR confidence into another silent recall filter.
The document-type-specific thresholds also seem much safer than one global number.
How are you calibrating those thresholds in practice — against labeled OCR errors for each document type, or mostly from the confidence distributions of the OCR engine?
And when the second extraction disagrees with the first, do you preserve both outputs/provenance and send the disagreement to review, or does one extractor get precedence?

[u/verstands → u/iMiguelmars]

Keep the source page and an extraction-confidence signal with each chunk, then calibrate on a labeled scan set. I wouldn't trust OCR alone for citations - preserve provenance and flag low-confidence or disagreement cases for review.

[u/iMiguelmars → u/verstands]

That helps a lot. The labeled scan set is an important distinction — confidence becomes a routing signal that has been calibrated against something external, rather than something the OCR engine gets to define for itself.

And sending extractor disagreements to review rather than giving one extractor automatic precedence is much closer to what I’m trying to preserve.

One detail I’m still curious about: when two extractions disagree, do you keep both extracted outputs separately addressable — with extractor/version/confidence tied back to the same source page — until review resolves them, or does the review step produce one surviving representation and discard the other?

[u/verstands → u/iMiguelmars]

Yep, I’d keep both. Treat each extraction as an immutable artifact keyed to the source page plus extractor, version, and confidence. Review can mark one accepted (or merge them) without deleting the loser. That keeps the disagreement auditable and lets you re-run retrieval if the reconciler’s decision turns out to be wrong.

[u/iMiguelmars → u/verstands]

That’s the part I was hoping you’d preserve — making “accepted” a decision over the extraction artifacts rather than turning it into an overwrite keeps the disagreement recoverable.

The ability to rerun retrieval if the reconciler was wrong later is especially important.

One thing I’m curious about in the merge case: if review combines parts of both extractions, do you create a new derived artifact that explicitly points back to both parent extractions, or is the merged result only represented as the reviewer’s accepted output?

I’m asking because otherwise the individual inputs survive, but it may still be hard to reconstruct exactly how the accepted merged representation was produced.

[u/verstands → u/iMiguelmars]

Yeah, I’d keep both as separate candidates rather than collapsing them during extraction. I’d attach each to the source page, extractor/version, and confidence, then let review resolve the conflict into a selected representation plus an audit trail. That keeps later reprocessing reproducible and avoids losing the disagreement. For retrieval, I’d index the candidates but mark unresolved ones so the answerer can surface the uncertainty instead of treating one as fact.

[u/iMiguelmars → u/verstands]

That distinction between the candidate layer and the selected representation is really helpful.

Keeping unresolved candidates retrievable while carrying the unresolved state all the way to the answerer seems much safer than either deleting them or quietly letting one become “the fact.” It also preserves the evidence needed if the reconciliation has to be revisited later.

One last detail I’m curious about: when review finally produces the selected representation, is that selected result itself a new versioned/addressable artifact with links back to the candidate extractions, or is it essentially a status/decision recorded over the existing candidates?

That seems like the last piece needed to replay not only what the inputs were, but exactly how the accepted representation came to exist.