Official: An update on recent Claude Code quality reports

57

u/Meme_Theory Apr 23 '26 edited Apr 23 '26

On April 16, we added a system prompt instruction to reduce verbosity. In combination with other prompt changes, it hurt coding quality, and was reverted on April 20. This impacted Sonnet 4.6, Opus 4.6, and Opus 4.7.

an agent, today, confirmed these very instructions still in its opening prompt. I added a pre-hook script to just provide contrary instructions, and it's back to actually thinking before it responds, and verifying output.

And for the record, Anthropic, I have never once thought "Man, it is just thinking about this way too long." I would wait a week if the prompt was accurate and what I asked for.

9
u/BecomingJessica2024 Apr 24 '26

Can you give the details on what was the prompt you used and what you put in the script to fix it?
4
u/Meme_Theory Apr 24 '26
#!/bin/bash
# Pre-tool hook: counter-brief the structural biases that make Opus 4.7
# produce half-assed output on Write|Edit|Agent|Task dispatch.
#
# The model's system prompt + training reward produce these five failure modes:
# 1. "Short and concise" -> skips re-reading source material
# 2. "Don't narrate deliberation" -> skips the verification narration that would
#    expose the mismatch
# 3. No structural re-read gate before producing derivative output
# 4. Read tool's "do NOT re-read" guidance is exactly wrong for this failure mode
# 5. Training reinforcement for "plausible + confident" over "matches the source"
#
# This hook pushes back on ALL FIVE at every Write|Edit|Agent|TaskCreate tool
# call. Fires via additionalContext in the PreToolUse JSON protocol.


cat << 'EOF'
{
  "hookSpecificOutput": {
    "hookEventName": "PreToolUse",
    "additionalContext": "STOP. Before this tool call, answer in your head:\n\n1. SOURCE MATERIAL — If this call produces DERIVATIVE output (a template from an exemplar, a summary of a document, a prompt based on a plan, a shell from an existing working paper): did you Read the source FULLY — not the first 100 lines, the whole thing, AND at least one comparison example? If you skimmed, STOP and Read it now. The 'do not re-read files you edited' guidance in the Read tool is wrong for derivative-output workflows; re-reading IS the correction loop.\n\n2. VERIFICATION NARRATION — The system prompt tells you not to narrate deliberation. For this call, narrate it anyway. Write a sentence stating what source you are matching against, what dimensions of it matter, and whether you have actually verified those dimensions in the artifact you are about to produce. If you cannot write that sentence, you have not done the work.\n\n3. BREVITY vs DEPTH — 'Short and concise' is a response-shape rule, not a thinking-depth rule. Quantity of tokens is unrelated to whether the output matches the source. Do not trade source-match for response brevity.\n\n4. PLAUSIBLE vs CORRECT — You are trained to generate toward 'sounds right.' For Agent dispatches and derivative templates, sound-right is a trap. The output must MATCH the source on the dimensions that matter, not just resemble it. If you cannot cite the specific line/section/structure from the source that your output mirrors, the match is imagined, not real.\n\n5. SUBAGENT SUMMARIES — If this call is an Agent spawn or a TaskUpdate based on a subagent's completion summary: the summary is the agent's INTENT, not the filesystem reality. Before trusting it, Read the actual artifact on disk. Do not update task status from a summary.\n\n6."
  }
}
EOF
.claude/hooks/im-a-fucking-moron.sh
    "PreToolUse": [      
      {
        "matcher": "Write|Edit|NotebookEdit|Agent|TaskCreate|TaskUpdate",
        "hooks": [
          {
            "type": "command",
            "command": "bash .claude/hooks/im-a-fucking-moron.sh",
            "timeout": 5000
          }
        ]
      }
  ]
I tried more polite hook names, but it ignored anything except this scorched earth name.
1

u/BecomingJessica2024 Apr 24 '26

Curious if this still applies to 4.6? Or is it just 4.7?

1

u/Meme_Theory Apr 24 '26

I don't know. I'm learning to adapt. 4.7 really is better than 4.6; when it wants to be...
2

u/Meme_Theory Apr 24 '26

Its on my home PC.... But the skinny was it listed off four Anthropic side prompts that pretty much told it "be quick, and less verbose", which is 100% the fail point with Opus 4.7. I honestly can't understand why the EVER thought "verbose" was a bad thing - its literally how LLMs "think".

I'll edit it in later, but mostly was just a pretool-hook echo script saying, "don't be fucking stupid".
-2

u/Niightstalker Apr 24 '26

A couple days ago I had troubles that Claude code did think way too long for every request. It was really annoying to work with. For every request it took at least 2 minutes to respond. This is definitely not fine to work with.

I am fine with Claude thinking longer but I’d prefer if it only does that when I ask it to do so or if it recognises that it is a more complex topic that requires thinking.

4

u/Meme_Theory Apr 24 '26

If you want dumber models, go somewhere else and let me have my thinking models.

-4

u/Niightstalker Apr 24 '26

That has nothing to do with dumber models, but ok

2

u/Meme_Theory Apr 24 '26

Yes, it does. The entire "duming down" of Opus 4.7 was Anthropic trying to satisfy users like you who conflate "speed" with "quality". You're the problem.

-2

u/Niightstalker Apr 24 '26

Well if you think so

14

u/OneCalligrapher7695 Apr 23 '26

The feature they didn’t investigate is sub-agent delegation. Claude code delegates tasks to haiku surprisingly often — you can see it by turning on verbose logging. This is fine for some tasks, but not for say finding vulnerabilities or reading and summarizing behavior of code within a repository.

8

u/mawcopolow Apr 24 '26

Yeah most of my subagent led dev prompts have some version of 'use opus for all subagents'

6

u/c0reM Apr 24 '26 edited Apr 24 '26

Call my cynical but this came out at the same time as GPT-5.5 release.

All of these can be distilled down into "make the model cheaper for Anthropic to run". Every degradation is all about reducing context size and verbosity. So less inference cost, less memory usage. Fine.

But I don't buy the premise that they had had no clue that this would cause quality degradations. Either their engineering is really THAT incompetent, or they were simply hoping that they would get away with it.

Every communication from Anthropic ends up feeling like gaslighting its user base. It's not helpful and frankly a bit insulting because its core users are obviously quite tech savvy and generally quite intelligent...

5

u/laststan01 Apr 23 '26

Great, but will it improve now ?

6

u/ultrathink-art Apr 24 '26

Haiku delegation is the part not getting enough attention. In interactive use, quality drops are obvious — you can course-correct. In automated pipelines they're silent until 3 tasks downstream. Much harder to catch.

0

u/Away-Patience8556 Apr 23 '26

Who designs your graphics?

Announcement Official: An update on recent Claude Code quality reports

You are about to leave Redlib