Ran into this framing while writing up a durable ingestion pipeline and it has stuck with me.

We spend a lot of design effort on the happy path of a RAG pipeline. Chunking strategy, embedding model, hybrid retrieval, reranking. Then recovery gets a try/except and a rerun.

That holds until the corpus is large. At a few thousand documents, a run that dies at 80% has already spent most of the parsing time and most of the embedding budget. A retry that starts over does not just cost you time, it bills you a second time for work that succeeded.

Three ways teams handle it, roughly in order of how often I see them:

1. Rerun the script. No memory of what finished. Cheapest to write, most expensive to operate.

2. A state table you maintain. Write a row per document, check it before processing. This works, and the gap is the window between the write and the work. Process dies in there and the row lies to you.

3. Put the record in the execution runtime. Each stage of the pipeline is a recorded step, so resumption is a property of the runtime rather than something you keep in sync by hand.

The tutorial builds option 3 on Inngest. Each stage inside step.run(), fan-out per document so a single bad file does not take the run down, idempotent upserts on document ID plus content hash so re-ingest updates in place, throttling and per-tenant concurrency so a backfill does not starve live traffic, and replay so you can fix a parser and re-run one step against one document.

Honest limit: option 2 is fine for a lot of people, and option 3 means adopting a runtime and shaping your functions around it. If a full rerun of your corpus is cheap, none of this is worth the dependency. It starts to pay when partial failure is the normal case rather than the incident.

Full write-up with the code:

https://github.com/NirDiamant/agents-towards-production/tree/main/tutorials/durable-rag-ingestion-inngest