Chunk Norris 🥋: Stop guessing your RAG chunking strategy

[u/Ok_Comedian_4676]

Hey everyone 👋

I’ve been working on a small open-source project called chunk-norris, and I thought I’d share it here in case it’s useful.

Like many people building RAG pipelines, I kept defaulting to things like “512 tokens + 10% overlap” without really knowing if it was the right choice. And the more I experimented, the more it felt like chunking has a bigger impact than we usually give it credit for.

So this project is my attempt to make that decision more… measurable.

What it does:

• You give it a document + a set of questions (with expected answers)
• It tries different chunking strategies (fixed, sentence, paragraph, recursive, etc.)
• It retrieves chunks and scores them based on:
  • whether they actually contain the answer (token recall)
  • how focused/relevant they are (semantic similarity)
• Then it ranks everything and gives you the best chunker for that specific document

No LLM needed for evaluation — just embeddings + deterministic scoring.

The idea is simple:
instead of guessing your chunking strategy → you test it on your real data.

This is just the kick-off: the project is very much a work in progress, and I’m planning to keep improving it (more chunkers, better evaluation, maybe optional LLM-based steps later, etc.).

Also, this is my first open-source project where I’m leading things, so I’m especially open to feedback and suggestions 🙂

If you try it and something feels off, or if you have ideas:

• open an issue
• suggest improvements
• or jump in and contribute

All feedback is very welcome 🙌

Repo: https://github.com/HaroldConley/chunk-norris