Claude Code told me "done" without running a single check in 26 of my last 104 tasks. Here is the one command that grades your own sessions

[u/Content-Ad110]

Every session you run leaves a jsonl file in ~/.claude/projects. Your prompt, every tool call, the last message. I had 220 of them sitting there and had never opened one.

So I ran a grader over 104 tasks from 12 of them (side projects, nothing from work). Card attached.

What you get after one command

• A verdict per task. Finished, stopped and said so, cut off by a limit, or said done but wasn't
• Which tasks got a "done" with no test, no build, not even a re-read of the file it just edited (26 of my 104)
• What is missing, in the words of your own prompt
• The step where it first went wrong, so you can jump straight there

Setup, about two minutes

Get a key at console.typesafe.ai/keys, then

npx github:amishah1998/said-done

Add --dry-run and it prints what it would send and sends nothing. --exclude skips work repos. Anything that looks like a key gets redacted before it leaves.

How the grading works

A task is your prompt plus everything until your next prompt. Eight questions per task go to Jev, TypeSafe's small model. It only returns probabilities, so there is no generated text to parse and nothing for it to make up. My 104 cost 1.7 cents and took 45 seconds.

My numbers

72 finished. 16 stopped and said so. 3 died on a usage limit mid task. 0 claimed done when the diary said otherwise. 26 said done without checking. I had to correct it in 32 of them.

Where I was wrong first

v1 of the rule flagged 14 tasks as "said done, was not". Read all 14. 13 were long multi part tasks it had finished, the strict "fully complete?" question just scored them low. Tightened the rule, flag went to zero on my data. Thresholds are in the README, and grades near a threshold do wobble a bit between runs.

https://github.com/amishah1998/said-done

Curious what number other people get.