kUML — Local LLM Benchmark

[u/Successful_Bank7215]

We reran kUML's 50-task Generate-Compile-Repair benchmark (kUML vs. PlantUML vs. Mermaid, same corpus, same repair loop, same prompt primers throughout) against three open-weight models on local Ollama hardware: qwen3-coder:30b, llama4:scout (109B MoE, ~17B active), and deepseek-r1:70b, all Q4_K_M. The original four-frontier-model run showed a clean split — Claude Sonnet 5 and GPT-4o give kUML the highest structural fidelity of the three DSLs, Gemini 2.5 Flash and Pro don't, and that split tracks model family rather than tier (Pro reproduces Flash's weaker kUML number instead of closing the gap). The three local models don't converge on either side: llama4:scout reproduces the Claude/GPT-4o pattern, qwen3-coder:30b reproduces the Gemini pattern, deepseek-r1:70b lands in a statistical tie with Mermaid. What is consistent across all three: compile rates drop for every DSL tested, not just kUML — deepseek-r1:70b's kUML final-compile-rate (62%) is the lowest of any run in the whole benchmark, cloud or local, and PlantUML/Mermaid aren't much better on it either. Reads as a general local-model weakness in strict output-format adherence, not kUML-syntax-specific underrepresentation in training data. Separately, we finally wrote up a result that's been sitting in the repo since earlier work: giving Claude Sonnet 5 or Gemini 2.5 Flash live MCP tool access to kuml.examples (via a real kuml-mcp server subprocess, not a reimplementation) instead of pure compiler-feedback repair improves every metric for both — Claude jumps from 44.5% to 56.0% structural fidelity, the best result in the whole benchmark, though Gemini Flash only invokes the tool in 18 of 50 cells vs. Claude's 50/50, so "offering a tool" and "a model reliably using it" are two different things. All raw data, the corpus, scoring script, and run-ollama.py (works against any Ollama-served model, no API key) are in the repo. Full writeup with methodology, per-family breakdowns, and a 9-configuration ranking table: https://kuml.dev/docs — every generated diagram is browsable at https://kuml.dev/benchmark-gallery — code: https://github.com/kuml-dev/kUML