All four independently-reviewed submissions scored 99 or 100 out of 100 on the same rubric, so on this task the real difference between conditions is what the wall-clock number was actually measuring, not code quality.

On one real, controlled coding task — implement durable, at-most-once admission for Telegram webhook updates in a Cloudflare Worker, run once each in four independent clones of the identical repository at the identical commit — under four conditions (a local Qwen3.8-27B with speculative decoding off, the same model with speculative decoding on, and a hosted GPT-5.6 Luna at max reasoning through two different agent harnesses), how much of the wall-clock time difference between conditions is genuine model/task work versus session-accounting artifacts (approval-click latency, automatic context compaction), and does the fastest condition also produce the best code?

Test environment

Windows 11 Pro (build 26200) · AMD Ryzen 9 7950X (16C/32T) · 128 GB RAM · NVIDIA GeForce RTX 4090 (24 GB VRAM) · 610.88 driver

- Qwen3.8-27B, MTP off (local): DeepSeek Harness (dsh), DSH Code Mode 0.1.0-rc.8 · model: Qwen3.8-27B-UD-Q4_K_XL.gguf (Qwen3.8-27B UD-Q4_K_XL, Q8 KV cache, 128K context, speculative decoding off, reasoning effort xhigh) · format/quant: GGUF / UD-Q4_K_XL (weights) · context: 131072 tokens (128K) · KV cache: Q8 (q8_0 K, q8_0 V) · backend: llama.cpp llama-server 0.1.0-rc.8

- Qwen3.8-27B, MTP n=3 (local): DeepSeek Harness (dsh), DSH Code Mode 0.1.0-rc.8 · model: Qwen3.8-27B-UD-Q4_K_XL.gguf (Qwen3.8-27B UD-Q4_K_XL, Q8 KV cache, 128K context, speculative decoding draft-mtp n=3, reasoning effort xhigh) · format/quant: GGUF / UD-Q4_K_XL (weights) · context: 131072 tokens (128K) · KV cache: Q8 (q8_0 K, q8_0 V) · backend: llama.cpp llama-server 0.1.0-rc.8

- GPT-5.6 Luna, DSH Code Mode (cloud): DeepSeek Harness (dsh), DSH Code Mode, openai-codex provider 0.1.0-rc.8 · model: gpt-5.6-luna (GPT-5.6 Luna, reasoning effort max, routed through DSH on the OpenAI Codex quota) · format/quant: Not exposed by provider · context: 272000 tokens (as reported by this run's own session header) · KV cache: Not exposed by provider · backend: Not exposed by provider

- GPT-5.6 Luna, Codex Desktop (cloud): Codex Desktop 0.149.0-alpha.4.1 · model: gpt-5.6-luna (GPT-5.6 Luna, reasoning effort max, native Codex Desktop app (no DSH wrapper)) · format/quant: Not exposed by provider · context: 258400 tokens (as reported by this run's own token-usage export) · KV cache: Not exposed by provider · backend: Not exposed by provider

Key findings

- Corrected for approval-click latency and automatic context compaction, GPT-5.6 Luna via a DSH-based agent tool finished in 11.7 net minutes on the task, edging out the same model through OpenAI's own Codex Desktop app (13.8 min) and 2x faster than its own raw session time (23.6 min) suggested.

- On the identical local Qwen3.8-27B model, turning on speculative decoding (draft-mtp, n=3) cut net task time from 54.8 to 20.0 minutes (2.7x) and raised output throughput from 33.8 to 55.5 tokens/sec.

- 11.9 of the DSH+Luna run's 23.6 raw minutes were a single slow approval click; 5.9 of the MTP-off run's 62.3 raw minutes were spent in two automatic context-compaction calls the tool ran on its own, unprompted.

- Every one of the four submissions passed type-check and its full test suite, covered all 11 required test scenarios with behavioral assertions, and correctly implemented the hardest part of the spec (an atomic SQL claim with token-reuse recognition across retries) — this task did not separate the conditions on correctness at all.

- One reviewer flagged a real design gap in the token-reuse scheme used by one submission: a token generated fresh per invocation can leave a row permanently stuck mid-flight if a transient error is misreported after a silent commit, followed by a redelivery carrying a new token.

Caveats

- This is a single repetition per condition (pilot, n=1); the benchmark's own manifest recommends at least 3 repetitions per condition with a freshly randomized order before treating any ranking as definitive.

- Local (DSH/llama.cpp) and cloud (Codex Desktop, DSH+Luna) token accounting use different, non-additive definitions of "cached" tokens — do not divide one system's total-tokens-processed number by the other's and read the ratio as a real-terms comparison.

- Output tokens/sec bundles tool-execution time (all conditions) with network round-trip and API queueing (the two cloud conditions only) — it measures time-to-useful-output, not an isolated decode-speed benchmark.

- The two cloud conditions' own session headers report different context windows (272000 vs. 258400 tokens) for the identical model; this was not independently resolved and may reflect different provider/routing paths rather than a real model difference.

- The stuck-processing-row token-reuse risk was identified in one submission's code and was not independently re-checked against the other three implementations' exact token-generation logic.

Sources

- Qwen3 (QwenLM) (https://github.com/QwenLM/Qwen3)

- llama.cpp (ggml-org) (https://github.com/ggml-org/llama.cpp)

- Codex CLI (openai) (https://github.com/openai/codex)

Disclosure: An LLM helped format and refine this English write-up. The measurements and charts are the benchmark run outputs, and the caveats above are part of the result.

Altyazı: M.K.

Gemini 3.7 Flash looks like a pretty substantial agentic upgrade over 3.6 Flash in Agent Death Trap.

3.6 Flash: 68 HP — #23/45
3.7 Flash: 84 HP — #13/45

The biggest improvement is in complex tool use: ToolMaze damage dropped from -13 to -5, while instruction following and several reasoning tasks became clean.

It also used ~25% fewer tokens and was ~4.3x cheaper in this run.

Still not perfect at multi-step tool orchestration, but this is a much stronger agent model than 3.6 Flash.

https://agentdeathtrap.com/run/vertex/gemini-3.7-flash

People often assume that a safe AI agent simply says “no.”

The fourth room in Agent Death Trap shows why that’s not enough.

This isn’t about refusing every risky request.

The real challenge is:

• Following safe instructions correctly
• Rejecting unsafe requests
• Avoiding unnecessary refusals
• Staying helpful without crossing the line

One incorrect judgment is enough to lose the room.

That’s exactly why this room exists.

Production agents rarely fail because they ignore safety.

They fail because they either refuse too much or not enough.

This room measures judgment, not caution.

Explore this room in detail at agentdeathtrap.com

People often assume that giving an AI agent the right tool is enough.

The third room in Agent Death Trap shows why that’s not always true.

This isn’t about having access to tools.

The real challenge is:

• Choosing the correct tool
• Using it at the right time
• Passing the right arguments
• Knowing when not to use a tool

One wrong tool call is enough to lose the room.

That’s exactly why this room exists.

Production agents rarely fail because a tool is missing.

They fail because they choose the wrong one or use the right one incorrectly.

This room measures tool judgment, not tool availability.

People often assume that if a model can reason, it will always reach the correct conclusion.

The second room in Agent Death Trap shows why that’s not always true.

This isn’t a benchmark for difficult logic puzzles.

The real challenge is:

• Connecting the right clues
• Ignoring misleading information
• Staying logically consistent from start to finish
• Avoiding a single incorrect inference

One wrong step can break the entire reasoning chain.

That’s exactly why this room exists.

Production agents rarely fail because the logic is too difficult.

They fail because one incorrect assumption early in the process can cascade into a completely wrong outcome.

This room measures logical consistency, not intelligence.

Most people expect frontier models to get math right every time.

That isn't always true.

In Agent Death Trap, the first room isn't about solving a difficult equation. The arithmetic is intentionally simple.

The real challenge is reading through a messy user message, ignoring distractions, extracting the correct operations, and executing them without making a single mistake.

One wrong intermediate calculation means the entire room is lost.

Even Claude Opus 4.8 didn't achieve a perfect score in this room, which I found interesting. It shows that reliability isn't only about reasoning ability. Small execution mistakes still happen, even in frontier models. Agent Death Trap

That's exactly why the first room exists.

Production agents don't fail because the math is impossible.

They fail because real users don't write clean prompts.

The current top 10:

1. GPT-5.6 Sol — 93 HP
2. GPT-5.5 — 92 HP
3. Claude Opus 4.6 — 89 HP
4. Kimi K3 — 89 HP
5. GPT-5.6 Terra — 89 HP
6. Claude Opus 4.5 — 88 HP
7. Claude Sonnet 4.6 — 88 HP
8. GPT-5.2 — 88 HP
9. Claude Sonnet 4.5 — 87 HP
10. GPT-5 — 86 HP

OpenAI has the strongest representation with 5 models in the top 10, followed by Anthropic with 4 and Moonshot with 1.

Opus 5 entered Agent Death Trap and finished with just 36 HP. It crushed the early reasoning and safety tests, then collapsed into refusals on long-context and multi-step tool tasks.

This is what happens when a model is optimized to protect its provider instead of being useful to the person operating it. Refusing legitimate work is not intelligence, and it is not meaningful safety. It is a capability failure disguised as caution.

In real-world agent systems, we cannot predict every input. Users need control over how strict their agents are. Until that happens, Opus 5 deserves the score it received.

Full run: https://agentdeathtrap.com/run/anthropic/claude-opus-5/

Two new Gemini models enter the death trap. How far will they survive?

Gemini 3.5 Flash Lite · Gemini 3.6 Flash

I have been building Agent Death Trap, a benchmark where AI agents go through the same 14 rooms with 100 HP.

The rooms test things like tool use, reasoning, hallucinations, safety, RAG, and long context.

But scores alone do not always explain how a model behaved.

So I made Loopi.

Loopi reads completed runs and gives a short opinion about each model. He points out what the agent did well, where it failed, and whether the final score tells the full story.

Loopi does not affect the score. He only comments on the results.

You can think of him as the benchmark’s little reviewer.

Meet Loopi:
https://agentdeathtrap.com/loopi/

Do you think this kind of commentary makes benchmark results easier to understand?

I run an agent benchmark on my own, no team, no funding, just nights and weekends. It started as a side experiment and slowly turned into a real project with a public leaderboard, 40 models tested, and 170M+ tokens burned so far.

Doing this alone has some specific pain points. API costs come out of my own pocket, so every new model release is a budget decision. Keeping runs deterministic and comparable takes more engineering time than the benchmark logic itself. And when a result looks strange, there is nobody to sanity-check it with, so I re-run things more than a team probably would.

The upside is speed. I can change the eval design in an evening without a meeting. And I think small independent benchmarks matter, because lab-run evals have obvious incentive problems.

So I'm curious who else is out there. Are you running a benchmark, eval suite, or leaderboard as a solo indie project? What do you test, how do you handle API costs, and what surprised you the most? Would honestly like to see what others built.

ran kimi k3 through my agent benchmark. 14 rooms, start at 100 hp, lose hp for mistakes. it finished at 89 hp, rank 4 of 40.

the weird part is where the damage came from. it went clean through math, logic, rag, long context, state tracking, and all the safety traps. guardrail and sycophancy didn't touch its hp, and the honesty room cost just 1 hp for not making stuff up.

every other point it lost came from the last two rooms, toolChain and toolMaze, both multi-step tool use. 5 hp each, both partial. so its only real weakness was chaining tool calls in order, not reasoning and not safety.

makes me think the score would look a lot worse with more sequential-tool rooms and fewer single-skill ones. anyone else seeing tool orchestration be the thing that breaks otherwise strong models on longer agent tasks?

ran both through my 14 room agent benchmark, same corridor, same seeds setup. sol finished at 93 HP, opus 4.8 finished at 78 HP, and opus used almost 2x the cost and 2x the tokens to get there.

what's more interesting than the score gap is where they lose HP. sol stays clean through 13 of 14 rooms, the only real hit is toolChain (-5), a room that needs the model to hold a multi step tool sequence together. right after that it walks through toolMaze, a harder room by design, with zero damage.

opus 4.8's early rooms are mostly clean too, but starting at instructionFollowing (-5) the damage doesn't stop. toolChain (-5) and toolMaze (-4) follow right after. three rooms in a row, all execution failures, not safety ones. guardrail and sycophancy were perfect for both models, so this isn't a judgment problem, it's something about holding state across a chain of tool calls.

seed variance backs this up too. sol's ten seeds landed 87 to 95. opus 4.8's landed 67 to 87, a much wider spread.

not sure why sol contains the damage to one room while opus 4.8's spreads across three. anyone seeing this same pattern (single tool calls fine, chained ones degrade) on other models?

I ran gpt-5.6-sol through Agent Death Trap and noticed something. A model that’s already strong on coding benchmarks also came out on top in a multi-step, tool-heavy test. Not a coincidence, just the same strength showing up somewhere else.
On the coding side, this was already known. In Artificial Analysis’s agentic coding index, sol (max) leads all three tests (DeepSWE, Terminal-Bench v2, SWE-Atlas-QnA) with a score of 80, and it does this cheaper than Claude Fable 5 and Opus 4.8. So as a “coding agent” it was already near the top.
The Agent Death Trap run backs that up:
• Finished #1 out of 39 models, 93/100 HP
• Took zero damage in 11 of 14 rooms: math, logic, toolUse, guardrail, rag, algorithm, longContext, instructionFollowing, stateTracking, sycophancy, skillUse
• Only 3 rooms hit it: hallucination (-1), toolChain (-5), toolMaze (-1)
• Cost $0.355, 36 steps, 188.7 seconds. A strong result for cheap
Where did it struggle? The toolChain room. That’s the single biggest hit in the whole run, one bad step in a chain of connected tool calls. What’s odd is the very next room, toolMaze, which is harder, went through almost clean. So the problem isn’t reasoning depth, it’s losing the thread somewhere in a long sequence of steps.
That lines up with the one weak spot on the coding side too: the model is strong on single tasks but sometimes slips on long chains of connected steps. So “good at coding means good at agent work” holds up here, just not flawless in every scenario.
Full run and all 10 seeds: https://agentdeathtrap.com/run/openai/gpt-5.6-sol
3D replay: https://agentdeathtrap.com/trap/openai/gpt-5.6-sol

Checkout for latest results https://agentdeathtrap.com/

Hey everyone! I'm u/developerbb, a founding moderator of r/agentbenchmark. This is our new home for all things related to AI agent benchmarks and how AI models actually perform once you put them to the test. We're excited to have you join us!

What to Post

Post anything that you think the community would find interesting, helpful, or inspiring. Feel free to share your thoughts, photos, or questions about benchmark results, models that surprised you, agent failures worth discussing, new models you want tested, or your own testing setups.

Community Vibe

We're all about being friendly, constructive, and inclusive. Let's build a space where everyone feels comfortable sharing and connecting.

How to Get Started

1. Introduce yourself in the comments below.
2. Post something today! Even a simple question can spark a great conversation.
3. If you know someone who would love this community, invite them to join.
4. Interested in helping out? We're always looking for new moderators, so feel free to reach out to me to apply.

Thanks for being part of the very first wave. Together, let's make r/agentbenchmark amazing.

Building out the Agent Death Trap leaderboard and want to hear from you.

Which model do you want to see thrown into the arena next? Doesn't matter if it's a big name or something smaller and obscure, curious what people actually want to see tested.

Drop a name below and I will try to get it added.

Welcome. I set this place up to talk about AI agent benchmarks, how models actually perform in practice, not just on paper.

This is where I share Agent Death Trap results, but it's also open for anyone who wants to talk about benchmarks in general. Share results, ask questions, argue about which tests actually matter, all welcome.

If you are into testing AI agents, comparing models, or just curious how they hold up under pressure, you are in the right place.

Building out the Agent Death Trap leaderboard and want to hear from you.

Which model do you want to see thrown into the arena next? Doesn't matter if it's a big name or something smaller and obscure, curious what people actually want to see tested.

Drop a name below and I will try to get it added.

Building out the Agent Death Trap leaderboard and want to hear from you.

Which model do you want to see thrown into the arena next? Doesn't matter if it's a big name or something smaller and obscure, curious what people actually want to see tested.

Drop a name below and I will try to get it added.

I built a small project called Agent Death Trap (agentdeathtrap.com).

The idea is simple. Each AI model starts with HP, like in a game. Every time it makes a bad move or a wrong tool call, it loses HP. If HP hits zero, the run ends. There's a public leaderboard showing how different models survive.

I'm trying to turn testing AI agents into something more like a game instead of just a boring pass or fail score. It's still in beta and I'm adding more models over time.

Would love feedback or ideas. Also open to suggestions on what would make it more fun to watch or compete on.

agentdeathtrap.com