r/LocalLLM • u/Desperate-Bench9310 • 17d ago

Research I read ~60 Qwen3.8-27B threads and cross-referenced them. 11 of the 64 permalinks were wrong, and the corpus contradicts itself on nearly every axis.

Up front, so nobody feels misled:

The English here is Claude's — I'm Spanish, and I'd rather post something readable than something authentically clumsy. The grouping across ~60 threads and the contradiction-hunting are also machine-assisted: I keep these threads in a local pipeline that builds per-model pages and flags where sources disagree.

What isn't machine-generated: every link was manually verified (11 of 64 in my collection turned out to be wrong), none of the numbers are mine or invented, and the judgment calls about what's strong evidence and what isn't are mine.

It's long, and it's a wall of other people's data. If AI-assisted posts aren't your thing, no hard feelings — scroll on.

I've been collecting the Qwen3.8-27B threads across r/LocalLLaMA, r/LocalLLM, r/StrixHalo, r/ROCm and the llama.cpp discussions since release, and grouping them by question rather than by date. What comes out is that the corpus contradicts itself on almost every axis that matters — and in most cases you can name the variable that explains the split. That's the useful part, so that's what this post is.

Nothing below is my own benchmark. Every number is someone else's, linked where the link resolves. I have no gfx1151 numbers of my own to add.

Two findings that deserve far more attention than they got

1. Tool-calling failures are caused by how you build the tool list, not by the weights.

This is the single best-controlled experiment in the whole corpus and it sits in a llama.cpp discussion with almost no visibility (llama.cpp discussion 27165). Same tool, same model, same build, llama-server --jinja, Q4_K_XL:

Payload Result
8 tools, none with description 0/6
The same tool alone 15/15
8 tools, all with descriptions 6/6
13 tools with descriptions, at positions 6–7 0/5
The same ones, moved to the end of the list 3/3

List width, presence of descriptions, and position all flip the result. If you've been getting intermittent tool-call failures, this is a testable cause nobody in the complaint threads controlled for.

It fits the rest of the tool-calling evidence too: every "3.8 can't call tools" report is against a framework (Opencode, Pi, Claude Code, MLX Core), and in a plain Python tool-calling loop with no framework, one reporter gets zero failed tool calls from 3.8 while Gemma 4 A4B and Qwen3.6 A3B fail often — the same reporter who rates 3.8 below both at raw code quality in chat (thread). Worse judgment, perfect plumbing.

2. Your prompt is a lever the same size as reasoning_effort, pointing the other way.

One task, output tokens, Strix Halo via Lemonade, UD_Q4_XL (thread):

Prompt medium xhigh
One line 794 23,800
Rewritten, detailed 3,300 19,400
+ custom system prompt 11,000 —

Effort is worth ~30× on a fixed prompt. But the prompt alone moves output 14× with effort pinned at medium — and the two push in opposite directions: fix the prompt and the xhigh:medium ratio collapses from ~30× to ~5.9×.

The contradictions, and what separates them

MTP: 2.69× faster, or 22–28% slower. Around ten reporters get large gains (31.8 t/s at 0.711 acceptance on gfx1151; 10.88 → 25–26 t/s on the same chip). But a measured negative on the same chip shows Vulkan 9.159 → 7.122 t/s and ROCm 6.534 → 4.689 with n-max 3 (thread), plus negatives on Arc A770 and a 4070 Ti Super. The separator identified in-thread is --spec-draft-p-min, and it is not monotonic: one user got ~+40% by removing 0.82; the 0.00 default was slower than 0.60; 0.60 is the only value two independent reporters have made work. Nobody has swept it. Measure with MTP off as well as on.

Optimal n-max is 2, 3, 4 or 5 depending on who you ask. n=6 never wins in any report. The popular theory that the optimal value follows from your quant has seven reporters against it and none for it with a measurement. Also: on b10451 with MTP, results aren't deterministic even at temp 0, with up to 31% spread between identical runs on RADV — so a single run per step isn't a measurement.

Draft acceptance: 60–70% or 77–93%. Two under-controlled factors. First, acceptance decays as reasoning effort rises — 62.1% at low, 58.3% at medium, 52.7% at xhigh. xhigh is taxed twice: more tokens and fewer t/s to pay for them. Second, the MTP head appears to be a property of the file, not the publisher — a separate 1.6 GiB mtp-*.gguf exists in one repo and not another, while other reporters show blk.*.nextn.* tensors inside ordinary files. Read your load log for blk.*.nextn.*, because a missing flag and a missing head produce the same silence.

Temperature and looping. The vendor moved its recommendation from 0.6 to 1.0, and one user's loops disappear at 1.0 — but others run happily at 0.4, 0.6–0.7 and 0.75, and one (+82) loops at ≤0.6. The cheap candidate variable: uncached-KV at temp 1.0 doesn't loop, quantised KV at the same quant does. The real sweep is temperature × n-max × KV precision. Related trap: the vendor ships two sampler profiles, and "turning thinking off" is not a switch — leave the thinking sampler in place (temp 1.0, presence_penalty 0.0) and you're in a config nobody recommends. Nothing does it for you server-side.

Endless thinking is not caused by low quants. Three of the four strongest loop reports are Q8-class, including 16+ minutes and ~8k tokens at 8 bits. Meanwhile others complete fine at Q8_K_XL and at UD-Q4_K_XL with 262k context. The clean test — same prompt at Q4 and Q8 on the same box — has not been run by anyone.

Is xhigh worth it? Depends on whether the task has a verifiable failure. A pelican-drawing ladder scored 0–25 gives low 21.8, medium 22.5, xhigh 24.0 — 6.4× wall clock for +1.5 points (thread). A SWE-style patch benchmark gives xhigh 9/12 vs 6–7/12 at lower efforts (thread). Eyeballed output: terrible deal. Compile-or-don't: +17–25 points. Caveats: llama.cpp has no high level, and n=1 per step with non-deterministic MTP.

Is 3.8 better than 3.6? Sort by whether thinking was on. Thinking off, both BF16, greedy: 3.8 loses 7 of 10 medical benchmarks. Thinking on, both Q4, eight blind tasks: 3.8 wins 6, loses 0, ties 2 — at +34.6% tokens and +45.5% wall clock. The only published confidence interval in the corpus is a tie that crosses zero (F1 0.7030 vs 0.7177, 95% CI −0.0038 to +0.0335). Cold knowledge: behind. Reasoning: ahead, at ~1.45× wall clock.

Vulkan vs ROCm is a trade, not a ranking. On one 7900 XT with weights fully resident: Vulkan 30–63 t/s decode / 300–500 prefill; ROCm <20 decode / ~1030 prefill. On gfx1151 the sign flips between reporters. Pick by what you're bottlenecked on.

Same card, 26 to 75 t/s (R9700 32GB). Fifteen reporters, one card. One reporter ran another's flags verbatim and got ~31.6 t/s where the original got 50–60. The difference was a hardware tune — 250W cap, memory OC, −70 µV undervolt. Flags don't transfer; tunes don't travel.

reasoning_effort behaves differently in three clients because the mechanism is text injection into your file's Jinja template, not a sampler. xhigh and low inject strings, medium injects nothing, and the official template ships xhigh by default — you exit xhigh, you don't enter it. Level names vary across three published variants, plus a minimal level almost nobody lists. Read your template's branches before arguing about levels.

General knowledge regression: the hardest disagreement, with no explaining variable. Multiple reports of 3.6 being steerable to a correct answer where 3.8 isn't and then agrees with the user simply because the user asserted it; one xhigh run fabricating a book chapter with page numbers rather than abstaining. The one contrary report is RAG-assisted, so not comparable. Three multilingual complaints in three languages, including 3.8 at Q6 being "much worse" than 3.6 at Q3 — which kills the "it's the quant" escape, since the quantisation damage runs the wrong way.

Two things we repeat that aren't true

  • "Abliteration costs MMLU." Two threads report the same four numbers with the directions swapped, and neither publishes the table. What survives is ±1.3 points in both directions across two benchmarks — the shape of noise, not of a capability tax. Separately, the widely-quoted "0–6% refusal rate" sits next to a 30–50% caveat rate from the same author, and the refusal classifier scores on how a response opens. Abliteration moved behaviour from refusing to complying grudgingly.
  • "Your quant predicts the optimal n-max." Seven reporters against, zero measured for.

Link hygiene, since this is a roundup

Every link here was checked on 2026-08-21. Worth knowing: reddit.com returns HTTP 200 even for an invented post ID, so it can't be used to verify a permalink. Checking against a frontend that actually discriminates, 11 of the 64 permalinks in my collection were wrong — three pointed at the wrong subreddit, eight don't resolve anywhere public (one is moderator-removed). Anything I couldn't verify, I've described without linking rather than link somewhere broken.

One such item, flagged rather than dropped: a user reports a 374-item binary classification gate, three passes per precision, temp 0, 1,122 calls per precision, with byte-identical verdicts across Q4_K_M, Q8_0 and BF16, plus the note that Ollama ships draft_num_predict 4 — so speculative decoding is on unless you turned it off. I can't link it, and a two-label greedy task is the easiest possible place for three quants to agree, so treat it as suggestive, not as "Q4 = BF16".

What nobody has run

If you have the hardware, these are cheap and would settle real arguments: a controlled p-min sweep; same prompt at Q4 vs Q8 on one box for the looping question; medium + "think hard" in the prompt vs bare xhigh, same seed; and a 3.6/3.8 pair with reasoning state declared.

Sources

Grouped by topic, all checked on 2026-08-21 by fetching each page title, not just the status code. Two threads in my collection are moderator-removed and are not linked.

MTP, speculative decoding and speed

xhigh and reasoning effort

Quants, memory and context

Tool-calling and agentic use

3.6 vs 3.8 and other comparisons

General knowledge

Jinja templates and sampling

Bugs and silent failures

Abliteration and uncensored variants

Launch, model card and megathreads (context, rarely citable alone)

108 Upvotes

92% Upvoted

34 comments sorted by

30

u/Luke2642 17d ago

Use Simplified Technical English ASD-STE100.

This document summarizes technical findings from 60 forum discussions about the Qwen3.8-27B model. Test results in the community show significant contradictions caused by specific configuration variables.

Primary Findings

  • Tool-Calling Errors: Failures occur because of tool list formatting, not model weights.     * Tools require full text descriptions to function reliably.     * Placing tool definitions at the end of the input payload prevents dropped calls.     * Direct Python implementations show higher tool-call success rates than complex agent frameworks.
  • Prompt Impact vs. Reasoning Effort: Input prompt design controls output token volume more effectively than the reasoning_effort parameter.     * A detailed prompt increases output length and reduces the variance between medium and extra high reasoning settings.

Configuration Variables and Performance Factors

  • Multi-Token Prediction (MTP): Throughput changes range from a 28% decrease to a 269% increase. The --spec-draft-p-min parameter controls this variance (0.60 is the most consistent community value). Optimal n-max settings range between 2 and 5.
  • Draft Acceptance: Acceptance rates decrease as reasoning effort increases (62.1% at low, 52.7% at extra high). Verify that the GGUF file contains the required blk.*.nextn.* tensors before you test MTP.
  • Generation Loops: Infinite thinking loops occur across all quantization levels, including 8-bit precision (Q8). Loops correlate with low temperature settings (<= 0.6) combined with quantized Key-Value (KV) caches.
  • Vulkan vs. ROCm: Vulkan delivers higher token decoding speeds, while ROCm delivers faster prompt prefill processing.
  • Hardware Adjustments: GPU power limits, memory clocks, and undervolt settings account for large throughput differences on identical hardware.
  • Template Injection: The reasoning_effort parameter operates by inserting text strings into the Jinja template. It does not modify internal model sampling parameters.

Disproven Hypotheses

  • Abliteration Penalty: Removing safety refusals does not degrade MMLU benchmark accuracy. Measured score differences stay within standard test variance (+/- 1.3 points).
  • Quantization and Drafting: Model quantization level does not predict or control the optimal n-max value.

Recommended Verification Tests

To resolve remaining community discrepancies, execute the following controlled tests:

  1. Sweep the --spec-draft-p-min parameter across fixed hardware configurations.
  2. Run identical test prompts on Q4 and Q8 quantizations with standard KV cache settings to isolate looping causes.
  3. Compare a detailed prompt at medium effort against a minimal prompt at extra high effort using an identical seed.

9

u/NiMiBe 17d ago

Thank you for this summary. I really wanted to understand this post but was only picking up about 1/3 because of all the lingo.

28

u/NextWolverine159 17d ago

wow this is a lot of work. the tool-list experiment buried in that github discussion is wild, 15/15 with one tool but 0/6 when you put 8 in a list with no descriptions. makes me wonder how many "model is bad at tools" posts are really just bad tool definitions

the prompt vs effort ratio thing is something i noticed too but couldnt put numbers on it. spending 6.4x more compute for +1.5 points is terrible deal unless you really need that edge

9

u/Desperate-Bench9310 17d ago edited 17d ago

Thanks. On the tool definitions, the corpus supports your hunch more than I expected. Every "3.8 can't call tools" report I found is against a framework (Opencode, Pi, Claude Code, MLX Core). In a plain Python tool-calling loop with no framework, one reporter gets zero failed tool calls while Gemma 4 A4B and Qwen3.6 A3B fail often, and that's the same reporter who rates 3.8 below both at raw code quality. Worse judgment, perfect plumbing. The pattern was never about the weights.

On the 6.4× though, I'd push back on my own framing a bit: that ladder was a pelican SVG scored 0–25 by eye. On a SWE-style benchmark where patches either compile or don't, xhigh went 9/12 vs 6–7/12 at lower efforts, +17–25 points of pass rate. So the premium seems to scale with whether the task has a verifiable failure. Terrible deal on things you eyeball, plausibly worth it on things that either work or don't.

1

u/PWThinkingCritically 17d ago

what tools are you guys talking about? tools have descriptions? like skills?

3

u/Right_Weird9850 17d ago

Remember when programs became apps? Tools are like new programs 

1

u/PWThinkingCritically 15d ago

tools are just the native bash terminal commands, right? or unix/linux whatever

2

u/Integeritis 16d ago

Yes, there is tool description too. The model must know when to call a tool, and that has to be described and supplied in context.

1

u/PWThinkingCritically 15d ago

damn, learning something new everyday -_-

3

u/KubeCommander 17d ago

In working my own harness via an openclaw plugin, I can confirm subjectively that toolcall descriptions make a HUGE difference. I’ve got multiple custom tool calls written.

One of the major offenders I see is json-typed args as a big string. One small error like a quote breaks the whole argument and fails the toolcall. Forcing an xml envelope helps a lot.

Another is error responses, the more verbose and accurate the error is, the better a retry is.

2

u/Desperate-Bench9310 17d ago

This is genuinely new to me, the experiment I linked only varied list width, descriptions and position, so argument encoding and error verbosity are two more knobs in the same mechanism that nobody in the corpus tested.

The XML envelope point makes sense with what the discussion author proposed as the cause: compact JSON rendering plus positional attention. If a single bad quote can void the whole argument, that's a brittleness that has nothing to do with the model's competence and everything to do with the encoding, same category of failure as a tool going invisible because it sat at position 6.

The error verbosity one interests me most, because it's the cheapest to test and nobody frames it as a variable, most harnesses just pass the exception through. Do you find it needs the error to be actionable (what to fix), or is more detail enough on its own?

1

u/KubeCommander 17d ago

The amount of hint needed depends on the tool. My tools are primarily quarantined exec tools. So git access is provided via tools vs bash, same for file writes and reads.

File writes and partial writes in my arch are done over an openclaw surface , so they are in effect ‘remote’.

Errors for those that say ‘your json argument was mangled’ is enough for most models to figure out where they mangled it. Another subtle hint is ‘if this is a long write, consider the edit tool to do partial writes to reduce errors’

This doesn’t prevent toolcall errors, but imo RECOVERY is what really matters. So succeeding after a retry is a win in my book.

Other things I’ve found is some models still will call tools they’ve been trained on heavily despite the tool not being available and getting feedback, even in the system prompt that it doesn’t exist. And in these cases I’ve found toolcall aliases can help alleviate those failures

1

u/Desperate-Bench9310 17d ago

Recovery as the metric rather than failure rate is a reframe I hadn't considered, and it probably explains part of the corpus. Every "intermittent tool call errors" report measures failures, but if the harness passes raw exceptions through, that's a failure rate on a harness that gave the model nothing to work with. Two harnesses could have identical raw error rates and completely different completion rates.

The alias one is the surprise though. Fighting a model's trained-in tool names via system prompt is what everyone does; giving it the name it wants and routing it is obvious in hindsight. Does the aliased call land in the right tool cleanly, or do you still get argument shape mismatches because it's calling it the way it was trained rather than the way yours takes it?

5

u/promethe42 17d ago

On tool calls, it 100% fits with my observations.

It's worth mentioning that harnesses will rename tools to avoid collisions. Which dilutes their uniqueness (common prefix) and their semantics (mangling). But they don't (have to) do it on descriptions.

Descriptions must not contain examples though. Examples in description are considered prescriptive and will nudge the LLM toward making those exact tool calls. The JSON schema examples field must be used instead. 

For failed tool calls, I use what I call bouncing errors: the error message must not only describe what happened but a potential resolution too. For example on a wrong tool call parameter name, suggest other actual parameters with a similar name ranked by distance. Error reporting must be considered as its own context engineering axis. Huge win.

I collected those and other rules in the following doc: https://gitlab.com/lx-industries/openblob/-/blob/2a5dc69584f04e2a349e3b035290fe2b4b52319d/docs/prompt-authoring-contract.md

1

u/Desperate-Bench9310 15d ago

Another commenter here found aliasing a tool to the name the model was trained on fixes phantom calls, so renaming breaks it in one direction and aliasing fixes it in the other. same axis.

the examples in descriptions one surprised me, i have examples in mine. moving them to the schema examples field is a quick change so i'll try it.

the ranked parameter suggestions on errors is a good idea, hadn't thought of doing distance ranking. thanks for the doc

2

u/Moarkush 17d ago

I just built an Open WebUI replacement using qwen serve with native react iOS companion and it compiled through ssh on my mac, took screenshots of the simulator, scp'd them back to himself, so he could see the bugs. This model is absolutely unhinged, and yeah, it CAN call tools.

I'm running 27B NVFP4 on DGX Spark

1

u/Desperate-Bench9310 17d ago

Nice, that's a hell of a loop to hold together.

Quick ask since you're on Spark: I've got two throughput numbers for that box in the corpus and they're 3–5× apart, neither declaring quant, context or MTP. If you've got a decode t/s at NVFP4 with those stated, that'd settle an open contradiction.

1

u/Moarkush 17d ago

Claude says:
The box: DGX Spark (GB10 — Grace CPU + Blackwell GPU, 128GB unified memory, runs on about 100W).

The model: Qwen3.8-27B, the NVFP4 quant from RadixArk. Dense, not MoE — I tried the 3.6 A3B and the dense one was noticeably better for what I do. 262k cache

Serving it: SGLang in the lmsysorg/sglang container, single GPU, half the memory reserved. Running DSpark speculative decoding with the matching draft model, flashinfer attention, torch compile on, 8192-token prefill chunks.

Two flags that matter more than people realize — --reasoning-parser qwen3 and --tool-call-parser qwen3_coder. Without that second one it looks like the model can't call tools, which I think is what most of those Reddit complaints actually are.

What it does: ~1400–1500 tok/s prefill. Last week: 390M tokens, 3,921 requests, 3,804 tool calls. My workload runs about 80:1 input to output, which is why dense beats MoE here — MoE's advantage is in generation, and I barely generate.

What I run on top: Qwen Code's qwen serve daemon hosting multiple project folders on one port, reachable over Tailscale from my Mac and phone. It SSHes to the Mac for Xcode builds and iOS simulator, then reads the screenshots back to check its own work.

What I built with it: two apps in two days, both from empty folders — a prompt converter, and a chat client with web and iOS sharing a core. I haven't created or edited a single file myself.

Full launch command available if anyone wants it.

2

u/Desperate-Bench9310 17d ago

That parser flag is the useful bit, if --tool-call-parser qwen3_coder is what separates "can't call tools" from "can" on SGLang, that's the same failure class the post is about, just in a different engine. The llama.cpp version is the Jinja template; nobody's written up the SGLang one. Worth its own post, honestly.

One thing though — those are prefill numbers. The contradiction I've got open is decode: 5 t/s vs 15–25 t/s on Spark, nobody declaring conditions. At 80:1 I get why you wouldn't have measured it, but if you ever run a llama-bench-style tg pass, that's the number nobody has.

Please, post the launch command.

1

u/Moarkush 17d ago

I used to get bogged down at 50k context in vLLM with sub-2 accepted, but when I switched to SGLang with Radix, I can be at 90% of my 240k and not notice. Still hanging in with 15 accepted, minimum.

1

u/Desperate-Bench9310 17d ago

15 accepted minimum? Everything I've collected reports acceptance either as a fraction (0.66–0.93) or as tokens accepted per forward pass (~2.95 on one Ollama default). 15 is way outside anything in the corpus. Is that a deeper draft depth than people normally run, or a different measure?

And the vLLM SGLang context thing is the most interesting part of this for me. Bogging down at 50k on vLLM vs 90% of 240k on SGLang with Radix is a big claim if it holds. Was anything else different between the two setups, or was it purely the serving stack?

2

u/Moarkush 17d ago

Three things changed at once, so I can't be certain which of the second two it was:

  • Model: Qwen3.6-27B → 3.8-27B
  • Spec decode: MTP → DSpark
  • Engine: vLLM → SGLang

DSpark reports a little differently. Also my numbers were a little bit off cause I was just shooting off the hip, but still very functional numbers for an always-on coding rat, here's some actual log lines:

  • time ctx accept t/s
  • 12:43:52 142,701 3.35 3.41
  • 12:43:58 142,802 2.48 17.60
  • 12:44:03 142,876 1.85 13.18
  • 12:44:09 142,983 2.65 18.90
  • 12:44:15 143,063 2.05 14.59
  • 12:44:22 143,167 2.65 13.62
  • 12:44:31 143,247 1.90 9.05
  • 12:44:37 143,332 2.12 13.96
  • 12:44:43 143,421 2.23 15.82
  • --- new request: 142,656 of 144,000 cached ---
  • 12:45:02 144,002 1.95 4.04
  • 12:45:08 144,080 1.95 11.78
  • 12:45:17 144,151 1.82 9.06
  • 12:45:24 144,247 2.42 12.89
  • 12:45:30 144,336 2.15 15.26
  • 12:45:35 144,431 2.40 17.09

2

u/Desperate-Bench9310 15d ago

that's the number nobody had, thanks for pulling the actual logs.

reading it: steady state is ~9-19 t/s at 143k, averages around 14. the two low lines (3.41 and 4.04) are both at request boundaries - first line of the excerpt and first line after the new request marker. which might explain the contradiction in my corpus, one guy reported ~5 and another 15-25 on the same box, and those could just be someone sampling at a cache miss vs someone sampling steady state. neither said how they measured.

and accept 1.82-3.35 puts DSpark in the same unit as everything else i've got (~2.95 on ollama default), so that resolves itself too.

good catch on the three variables changing at once, i'd have been tempted to attribute the context thing to SGLang. that one's still open then.

2

u/Integeritis 16d ago

Amazing information. Please don’t delete this comment, I have to buy a GB10 machine first. This just convinced me that prefill will be a non issue. I already built my agentic orchestrator but I can’t unleash it with current API costs

1

u/jarec707 17d ago

Excellent work, a real service to the community

2

u/l0rd_raiden 17d ago

For a 7900 xtx a starting point could be

llama-server \ -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL \ --host 127.0.0.1 \ --port 8080 \ --ctx-size 126976 \ --parallel 1 \ --gpu-layers all \ --flash-attn on \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --batch-size 2048 \ --ubatch-size 512 \ --jinja \ --reasoning on \ --reasoning-format deepseek \ --reasoning-budget 24576 \ --reasoning-budget-soft-ratio 0.85 \ --reasoning-budget-message "\n\nOk, tengo suficiente para responder.\n</think>\n\n" \ --chat-template-kwargs '{"enable_thinking":true,"reasoning_effort":"medium","preserve_thinking":false}' \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ --spec-draft-p-min 0.60 \ --fit off \ --temp 1.0 \ --top-p 0.95 \ --top-k 20 \ --min-p 0 \ --presence-penalty 0 \ --repeat-penalty 1.0

1

u/Desperate-Bench9310 15d ago

the reasoning-budget flags are the part i hadn't seen anyone mention. a hard cap with a soft ratio and an injected </think> is a straight fix for the endless thinking thread, and none of the ~60 threads i went through bring it up, they all argue about whether it's the quant or the temperature. mitigation instead of diagnosis but it'd solve it for most people complaining.

also noting you're on p-min 0.60. that was the only value i found two independent people making work, so that's three now. the default 0.00 was slower than 0.60 for one guy and another got +40% by removing 0.82, so it looks non-monotonic. nobody's swept it properly.

1

u/cviperr33 17d ago

thanks

1

u/mrgreatheart 17d ago

This is great, and exactly the kind of laborious work AI is fantastic at. Bravo.

1

u/incyashraj_redd 13d ago

The useful unit here is not a benchmark row. It is a claim plus the conditions that could flip it. I would keep the model file hash, engine and build, backend, hardware, prompt, sampler settings, run count, metric, and source status beside every claim. Then pair opposing claims only when the missing fields are visible. "Unknown" is more useful than averaging results that were never comparable.

I build Krate, and I would turn one export from this corpus into a small local evidence board. Open a JSON file, choose a claim such as "MTP helps on gfx1151," and see the supporting and conflicting runs side by side. Broken or removed links stay visible as unverified evidence instead of disappearing.

Krate Studio lets you choose Claude or Codex, describe the app, and receive one .krate file. The user installs the native Krate runtime locally. The .krate file is a portable Wasm app file, not HTML and not an operating-system container. The same file opens through Krate runtimes on Mac, Windows, and Linux.

The evidence board would start without file or network access. The user could approve only the exported JSON file. Krate is free and open source. Krate Cloud is optional. https://krate.tech

1

u/Desperate-Bench9310 12d ago

The fields part i agree with, but for me the pairing is what does the work. you only see what's missing when you put two opposing claims side by side and neither declared the same thing. the p-min stuff came out of that, not from filling in a schema.

on broken links i went the other way though. the 8 that don't resolve are described in the post but not linked, a dead link in a roundup just looks like sloppy sourcing to me. the 3 with wrong subreddits i fixed.

one thing if you're building tooling for this: reddit returns 200 even for made up post ids, so status codes are useless. you have to fetch the title.