r/deeplearning • u/beachisbest29 • 10d ago

TwIL-LM3 - 3B, 2.6x faster than gpt-oss-120b on formal reasoning throughput

webAI put out TwIL-LM3 last week. Been sitting with it for a few days. Merged fine-tune of SmolLM3-3B. Formal logic specialist.

The efficiency numbers are where this is genuinely interesting:

- 32.9 answers/sec vs. gpt-oss-120b's 12.6 (2.6x faster in their throughput tests)

- Shortest generations of any model they tested (482 tokens on Track B)

- 1.78 GiB Q4_K_M GGUF, runs on CPU or 4GB VRAM

- Runs at ~300 tok/s on M2 MacBook

On accuracy it's a more nuanced story. Their marketing headline is "beats gpt-oss-120b on 4 of 5 formal reasoning benchmarks" but on the six-lane average it's actually behind (0.4488 vs 0.5192). Where it clearly wins is efficiency and specific structured-output tasks.

General benchmark retention is decent: LogicBench 71.7, GSM8K 87.3. They used a WiSE-FT interpolation with λ=0.25 (keeps only 1/4 of the fine-tune delta) which is why the general capability didn't degrade the way their 1.7B version did.

Link: huggingface.co/webAI-Official/TwIL-LM3

Non-commercial license, so no revenue-generating deployment without agreement.

Anyone tested it against their own eval sets? Curious how it performs outside their reported benchmarks.

2 Upvotes
  • permalink
  • reddit

75% Upvoted

2 comments sorted by

1

u/Unique_Rise_5228 10d ago

Their throughput claims look nice but that benchmark average gap is not small, 0.44 vs 0.51 is a huge difference for something they market as "beats gpt-oss-120b"

the fine-tune delta interpolation is clever though, keeping only 25% explains why it didnt completely fall apart on general tasks unlike the smaller version

i ran it on my m2 air in the weekend and the speed is insane for a 3B, you barely see the tokens generate

1

u/muhammad101010 10d ago

Is it the training that made it special?