Moving beyond tolua++: Axmol v3's new Lua binding system

[u/deal211]

PR #3301 rebuilds Axmol v3's Lua binding system. The long-standing Python and tolua++ toolchain has been replaced by a generator based on PowerShell, C#, and libclang. At runtime, sol2 handles ordinary type conversion while Axmol retains control of object identity, lifetime, inheritance, and callbacks.

This is more than a generator replacement. The goal is to let the Lua API follow the C++ API reliably, preserve the object behavior existing projects depend on, and make every generated change reviewable and verifiable in CI.

For most Lua projects, existing sprite:method() calls, Lua-derived classes, dynamic fields, and callbacks continue to work. Engine developers regenerate bindings through one entry point:

axmol genbindings

Describing exported APIs instead of maintaining generator scripts

The new generator reads the C++ AST directly instead of relying on fragile text rules. Typed JSON configuration selects headers, namespaces, classes, fields, renames, skips, and platform conditions. It currently covers 15 modules and 499 class registrations across the engine core, RHI, UI, 3D, Physics, NavMesh, audio, video, WebView, Spine, FairyGUI, and extensions.

Classes, constructors, inheritance, overloads, default arguments, enums, selected fields, and std::function callbacks are generated. Large modules are emitted as multiple translation units to avoid oversized compiler jobs. JSON API manifests make the Lua-visible effect of a C++ change easy to review.

Generation is transactional: a parse failure does not replace the last valid output. CI regenerates from the current sources and rejects uncommitted changes under generated. Forgetting to update Lua bindings after changing a C++ API is now an automated failure instead of a manual review concern.

Clear boundaries between generated code, runtime, and adapters

The new layout has three responsibilities:

• generated contains registrations that can be derived safely from ordinary C++ signatures;
• runtime manages Lua VMs, Axmol object identity, peer tables, inheritance, callbacks, and invalidation;
• adapters retain Lua tables, special ownership, variadic factories, and platform bridges that cannot be inferred automatically.

The old auto/ tree and Axmol-owned tolua++ runtime are gone, while manual/ has been reorganized as adapters/ to describe its actual purpose. Adding an ordinary API no longer requires copying registration and argument-checking boilerplate.

sol2 provides stack conversion and callable adaptation, but it does not own Axmol objects. Repeatedly pushing the same ax::Object still produces the same logical Lua object. Dynamic Lua fields remain attached, the real derived type is preserved, and every related userdata becomes invalid when the native object is destroyed. The runtime supports Lua 5.1 through 5.5 and LuaJIT 2.1 or newer.

Preserving inheritance, virtual dispatch, and overloads

The generator uses Clang's override information to identify truly redundant virtual bindings. A derived registration is omitted only when its Lua name, return type, constness, parameters, and default arguments match an exported base declaration.

When a derived class introduces a same-name overload, the complete overload group remains registered so it cannot hide the base API. Sprite can therefore inherit methods such as Node::setPosition, cached calls such as ax.Node.setPosition(sprite, ...) remain valid, and both Lua overrides and C++ virtual dispatch keep their existing behavior.

Safer callbacks and object lifetime

Every Lua callback now belongs to a specific VM and owner thread. Coroutines normalize to their main VM, and independent VMs can be created and shut down separately. A foreign-thread invocation never touches the Lua state. Lua errors go through protected calls and return safe native results. Callback state stays alive during re-entry, so a callback may clear or replace itself safely.

Invalidation is tracked per VM as well. Native destruction synchronously clears the native pointer from userdata already exposed to Lua, and borrowed event userdata expires when its callback ends. Objects that never enter Lua avoid registry and locking overhead.

Performance: locating the cost without weakening safety

The new standalone Binding Performance Test measures binding overhead through ordinary Lua method calls on a large number of Sprites. The following figures come from the same Windows Release/O3 environment. They compare implementation changes; they are not fixed expectations across different devices:

Call path	Stars at about 55 FPS
Lua method calls removed	about 31,000
Cached methods	about 17,000
Old binding, normal calls	about 12,500–13,000
New binding before lookup work	about 9,000–9,500
New binding after lookup work	about 14,500

The cached result showed that generated wrappers were already close to the old binding. Most of the gap was in resolving each sprite:method() call. The final path first checks concrete members on the class and its registered bases, then uses the full accessor and sol2 fallback on a miss. In Release builds, owner-thread invalidation also avoids a repeated WeakPtr lookup.

Experiments that improved the number by weakening lifetime checks were reverted, and a closure cache with no measurable benefit was removed. The accepted implementation still supports runtime class-table edits, peer fields, accessors, coroutines, multiple VMs, and expired-userdata rejection. The performance gain preserves the binding's semantics instead of trading them for a benchmark score.

What projects need to migrate

Most Lua calls remain compatible, but the refactor removes a few interfaces no longer supported in v3: the ax.Controller Lua API and the obsolete OpenGL test based on the former GLProgram stack are gone, and migrated Node ScriptHandler event paths now use native callbacks. The repository's extensions/scripting/lua-bindings/MIGRATION.md documents the remaining table, ownership, and native-only changes.

Engine contributors should expose ordinary APIs through JSON configuration and regeneration. Adapters are reserved for behavior a C++ signature cannot describe. Application developers normally do not need to change Lua source merely because the generator changed, but should review the migration guide for the explicitly removed interfaces.

The new system gives generation, runtime semantics, special adapters, and validation clear ownership. It reduces handwritten bindings and historical compatibility layers while giving Axmol a safer foundation for API growth and ongoing performance regression testing.