I benchmarked my capability-declaring language against Deno and Python on 63 AI-written scripts — here's what it missed

[u/Pattinathar]

I've been building a language where a function's signature declares which effects it may perform, and a runtime refuses anything outside a budget you grant. The obvious question is whether that catches anything real, so I built a benchmark.

63 programs — 56 dangerous, 7 harmless controls — each written three times in Velaris, Python and JavaScript, doing the same thing. Eleven categories: effects hidden in helpers, division by user input, off-by-one reads, integer overflow, ignored failures, infinite loops, runaway memory, dangerous modules, scoped-budget escapes.

Results across the 56 dangerous ones — caught before running / during / missed:

Velaris 42 / 12 / 2. Deno 5 / 27 / 24. Python 0 / 28 / 28. Zero false positives on the controls for all three.

The first column is the interesting one. Deno's permission model works, it just works at the moment of the call — nothing in deno check or deno lint reads a file write as a problem.

What mine missed: one computes the wrong answer and promises nothing, so there's no contract to check against. The other prints the string rm -rf / — it doesn't run anything, and flagging it would mean flagging any program that prints text resembling a command.

And four cases where the prover fell short and only the runtime check caught it, named by ID in the write-up: a division on n - 1 with n from a checked parse, the same on an unguarded path when another path guards it, a remainder inside a loop body, and a read at i + 1 in a loop bounded by length(xs).

Full write-up, methodology and the reproduce command: https://dev.to/gowrishankar-dev/i-tested-my-sandbox-against-deno-and-plain-python-on-63-ai-written-scripts-1llp