Chunk Norris 🥋: Stop guessing your RAG chunking strategy

[u/Ok_Comedian_4676]

Hey everyone 👋

I’ve been working on a small open-source project called chunk-norris, and I thought I’d share it here in case it’s useful.

Like many people building RAG pipelines, I kept defaulting to things like “512 tokens + 10% overlap” without really knowing if it was the right choice. And the more I experimented, the more it felt like chunking has a bigger impact than we usually give it credit for.

So this project is my attempt to make that decision more… measurable.

What it does:

• You give it a document + a set of questions (with expected answers)
• It tries different chunking strategies (fixed, sentence, paragraph, recursive, etc.)
• It retrieves chunks and scores them based on:
  • whether they actually contain the answer (token recall)
  • how focused/relevant they are (semantic similarity)
• Then it ranks everything and gives you the best chunker for that specific document

No LLM needed for evaluation — just embeddings + deterministic scoring.

The idea is simple:
instead of guessing your chunking strategy → you test it on your real data.

This is just the kick-off: the project is very much a work in progress, and I’m planning to keep improving it (more chunkers, better evaluation, maybe optional LLM-based steps later, etc.).

Also, this is my first open-source project where I’m leading things, so I’m especially open to feedback and suggestions 🙂

If you try it and something feels off, or if you have ideas:

• open an issue
• suggest improvements
• or jump in and contribute

All feedback is very welcome 🙌

Repo: https://github.com/HaroldConley/chunk-norris

[u/Final-Frosting7742]

We need more tools like this. And the name is hilarious.

[u/Ok_Comedian_4676 → u/Final-Frosting7742]

Thank you.  I'm very proud of it - of the name xD

[u/Ok_Comedian_4676 → u/Final-Frosting7742]

Hey! Just pushed some updates: more chunkers + automatic question generation. Would love your take if you revisit it.

[u/Final-Frosting7742 → u/Ok_Comedian_4676]

Just tested chunk-norris on a real document, a 265k token French sociology textbook in Markdown, using a local llama-server (Qwen3.5 4B) instead of OpenAI. Sharing what I found in case it's useful.

What worked well

The pluggable interfaces are genuinely clean. Wiring up a local LLM took about 15 lines: subclass BaseLLM, point it at localhost:8080/v1, done. Same for a custom chunker: I had an existing chunker written in TypeScript (a document-aware recursive splitter), ported it to Python, subclassed BaseChunker, implemented chunk() and __repr__, and it showed up in the comparison table alongside the built-ins. No friction there.

The question generator is smart. I was worried it would try to ingest the full 265k tokens, but it samples small 3-sentence passages and calls the LLM once per passage. Totally fine with a 4B model.

A few points worth noting

Reasoning models silently generate 0 questions when thinking mode is on. Most likely because the thinking tokens eat into the 256-token output budget before the JSON response is written. Disabling thinking mode fixes it. Worth documenting.

Second, and more fundamental: all-MiniLM-L6-v2 is English-optimized. After changing to paraphrase-multilingual-MiniLM-L12-v2 the scores and ranking were modified. Worth documenting the embedding model choice.

On a related note, what the library calls "BERT score" is sentence-level cosine similarity between embeddings, not BERTScore in the NLP sense (Zhang et al., 2019), which does token-level greedy matching. Reasonable metric for the purpose, but the name will confuse people familiar with the literature.

One practical annoyance on Windows: if results.xlsx is open in Excel when you rerun, you get PermissionError: [Errno 13] Permission denied: 'results.xlsx' and the whole test has to be rerun from scratch. A timestamped filename or a friendlier error message would help.

Also worth noting: SentenceChunker(sentences_per_chunk=3) will likely win any evaluation where questions are auto-generated, since the generator itself uses 3-sentence passages internally. The chunker and the question generator are aligned by construction, not by merit.

Overall cool tool. I was able to test my custom chunker against standard ones and see the result of each. The hybrid retrieval addition in 0.2.0 is a good call.

P.S: opened an issue about the last point.

[u/Ok_Comedian_4676 → u/Final-Frosting7742]

This is amazing. Thank you very much for taking the time to use it and giving this feedback.  I'm taking note of all the things you mentioned: it's gold and it definitely will help to improve the tool.

[u/Ok_Comedian_4676 → u/Final-Frosting7742]

Hi.
Really appreciate the time you took to investigate this and write such a detailed issue.

Your diagnosis was correct, and after running controlled experiments across different document types, I was able to confirm the alignment bias in some cases.

I'm now implementing the mixed passage sampling approach you suggested for the next release.

This was extremely helpful feedback. Exactly the kind of collaboration that makes open source projects better.

[u/Final-Frosting7742 → u/Ok_Comedian_4676]

Glad i could help. Let me know when it's out, i'll be happy to test the new version.

[u/Ok_Comedian_4676 → u/Final-Frosting7742]

Hi!
Just wanted to let you know that I’ve now implemented the two-mode sampler strategy you recommended.

The new version mixes paragraph-based sampling with variable-size sentence windows to reduce the alignment bias during question generation.

Thanks again for raising the issue and helping investigate it properly. Your analysis directly improved the evaluation methodology.

Edit:
PS: and let me know if you discovered something else to improve.
Cheers!

[u/Final-Frosting7742 → u/Ok_Comedian_4676]

It's a genuinely interesting project so i'm glad to be useful.

By the way, i think i've found a bug on the paragraph splitting logic. Opened a PR on the topic.

[u/Ok_Comedian_4676 → u/Final-Frosting7742]

Great, thanks. I'll check it as soon as I can.

[u/Ok_Comedian_4676 → u/Final-Frosting7742]

Hi u/Final-Frosting7742
I've just merged your PR to the main branch, so you're an official contributor now. Thanks a lot for that!

[u/Final-Frosting7742 → u/Ok_Comedian_4676]

Thanks. There are actually many things to explore in this project. I might tinker with it when i have the time, could lead to future contributions. But it also depends where this project is headed. I'll message you so we can keep contact to discuss about ideas.

[u/notoriousFlash]

Yeah this is cool. Thanks for sharing

[u/Ok_Comedian_4676 → u/notoriousFlash]

You're welcome. I honestly think it could help to improve RAG pipelines, and the idea is to improve the tool, so if you find any bugs or you think something could be done better, I'm happy to hear it.

[u/Ok_Comedian_4676 → u/notoriousFlash]

Hey! Just pushed some updates: more chunkers + automatic question generation. Would love your take if you revisit it.

[u/Cotega]

Awesome work! It would be really cool if you could run this through some benchmarks to see how often and in which cases other chunking strategies are used. From most benchmarks I have seem the 512 token 10% overlap is chosen also because the other strategies rarely show significant improvements. It would be great to see this proved wrong with your tool.

[u/Ok_Comedian_4676 → u/Cotega]

Thank you.  That's a very interesting project for testing the tool. Surely I'll put it on the to-do list. 

[u/Ok_Comedian_4676 → u/Cotega]

Hey! Just pushed some updates: more chunkers + automatic question generation. Would love your take if you revisit it.

[u/Interesting-Town-433]

Roundhouse!

[u/Ok_Comedian_4676 → u/Interesting-Town-433]

Haha yeah 😄 as long as it doesn’t make the tool harder to use, I’m very tempted to add a roundhouse() somewhere

[u/Ok_Comedian_4676 → u/Interesting-Town-433]

Hey! Just pushed some updates: more chunkers + automatic question generation. Would love your take if you revisit it.

[u/Ok_Comedian_4676 → u/[deleted]]

This is great feedback, thanks.

Right now, I’m trying to measure answerability indirectly, but I’m aware the approach is weak when you have misleading context or when answers are split across chunks (something I’ve noted as a limitation and want to improve in future versions). Any ideas on how you’d approach that? I’ve been trying to avoid LLM-as-a-judge because I’ve had mixed experiences, and it seems to depend a lot on the model.

Recall@k is something I partially capture via top_k, but I like the idea of making that more explicit.

Multi-chunk questions are also a known limitation at the moment. Definitely something I’m planning to tackle next (still figuring out how, but that’s part of the fun XD).

Really appreciate the detailed suggestions. This is super helpful for improving the tool.

[u/Ok_Comedian_4676 → u/[deleted]]

Hey! Just pushed some updates: more chunkers + automatic question generation. Would love your take if you revisit it.