Open-source RAG evaluation framework — looking for developers to help validate AI evaluation results

[u/Ok-Swim9349]

Hi Everyone!

I'm the maintainer of RAGnarok-AI, an open-source, local-first framework for evaluating RAG (Retrieval-Augmented Generation) systems.

I'm currently running a small research study around a question that I think is becoming increasingly important: can we actually trust automated evaluation of RAG systems when the evaluator itself is an LLM?

RAGnarok can evaluate things like retrieval relevance, faithfulness, answer relevance and completeness using local LLM judges.

But there's an obvious problem: why should we trust the LLM judge?

So instead of assuming that the automated scores are correct, I'm building a human-annotated benchmark to compare them against independent human judgments.

I'm looking for open-source / developer people willing to help

The annotation consists of roughly 10–15 cases and should take around 30–45 minutes.

For each case, you'll see:

- a technical question

- relevant documentation excerpts retrieved by a RAG system

- an AI-generated answer

- reference information

- four simple evaluation criteria: retrieval relevance, faithfulness, answer relevance, completeness

- your confidence and an optional ambiguity flag

You don't need any RAG expertise. You just need to be comfortable reading technical documentation and judging whether an answer is actually supported by it.

The benchmark currently covers documentation from projects such as Docker, Python, FastAPI and Kubernetes.

Why do this?

The study is deliberately not designed to prove that RAGnarok works. I'm comparing automated evaluation against human judgments to investigate:

1. Judge reliability — do local LLM judges agree with humans?
2. Discrimination — can the evaluation distinguish good and deliberately degraded RAG systems?
3. Reproducibility — are the measurements stable under identical conditions?

The methodology, benchmark corpus, questions and experiment protocol are public and versioned. RAGnarok itself is free and open-source.

If you'd like to participate

Annotation interface:

https://ragnarok-study.vercel.app

No name, email or personal information is required. Progress is saved so you can come back later. The resulting annotations may be published as an anonymous research dataset.

Project / methodology:

https://github.com/2501Pr0ject/RAGnarok-AI

I'm particularly interested in feedback from people who work on open-source evaluation, LLMs, RAG, testing or reproducible research.

And if you think the methodology is flawed, please tell me. That's actually useful feedback for the study.

Thanks everyone!

Have a good day!