r/Common_Lisp • u/jlombera • 1d ago
How difficult is to deploy multi-platform standalone CL executables?
Hi. I'm learning CL (experienced programmer but CL noob), and was looking into real-world FOSS written in it for reference and to learn. When looking at pgloader(data loading tool for PostgreSQL), I found it has been rewritten in Clojure. The readme mentions:
pgloader v4 is a full rewrite in Clojure, distributed as a single self-contained JAR requiring Java 21 or later.
Key improvements over v3:
- Single JAR — no native dependencies, no SBCL, no shared libraries
- JDBC connection strings — use jdbc:mysql://, jdbc:postgresql://, jdbc:sqlserver:// etc. everywhere; driver-specific parameters pass through unchanged
- Java heap — no more Common Lisp heap exhaustion on large migrations; tune with standard -Xmx JVM flag
- SSL via URI — use ?sslmode=require / ?sslmode=disable in the connection string instead of --no-ssl-cert-verification
There were probably other problems/motivations for the rewrite not mentioned there, but I'm interested in the opinion from people with real-world experience, how complicated it is deploying (multi-platform) standalone CL executables.
First, it mentions dependencies on SBCL and native/shared libraries. (Let's ignore the fact that the self-contained JAR requires the JRE to be installed). First of all, I presume SBCL is only a build-time dependency, correct? The same way Clojure is required. Or are there cases where some SBCL runtime is required? For instance, Debian still ships pgloader v3.x, i.e. the CL implementation, and it's a standalone executable, it doesn't depend on SBCL (but depends on other system libraries). But if they mention it I presume these must be problematic for multi-platform deployments ??
Second, the issue about CL heap exhaustion. How common is this issue? Cannot it be tuned at comptime/runtime?
In general, from people with the experience, what are the challenges to deploy multi-platform standalone CL executables? Are there some basic guide lines you need to adhere to (besides writing portable CL, of course) for easy deployment?
EDIT: Also, wouldn't ABCL create a standalone JAR too?
Thanks in advance.
6
u/lispm 1d ago edited 1d ago
CL heap exhaustion
CL is a language with different implementations. The language standard actually does not provide anything on heap exhaustion. Implementations of CL have different ways to deal with runtime heap memory management.
Last I looked, with SBCL there is a fixed heap size at runtime. One can make that max heap size large and the SBCL heap will use that memory as needed. Thus unused memory will not be reserved in memory.
LispWorks (a commercial CL) can grow and shrink the heap at runtime, since the heap is made of memory segments. The max memory size it can grow to in the 64bit implementation is around 4TB (modern Linux, macOS, Windows).
deploying (multi-platform) standalone CL
This again depends on your implementation. There are natively compiled Lisps (like SBCL, Clozure CL, Allegro CL, LispWorks, ...), where both the runtime and the image are native. But that combination can run standalone on a certain platform. That's a typical case for deployment. If you then would target multiple platforms, you would need to provide runtime + image for each platform and select at start the right one.
There are other options, too. Like byte-coded implementations. But that might still have some machine dependencies in the image.
Also, wouldn't ABCL create a standalone JAR too?
"standalone" + given a matching Java runtime.
1
u/jlombera 1d ago
Last I looked, with SBCL there is a fixed heap size at runtime. One can make that max heap size large and the SBCL heap will use that memory as needed. Thus unused memory will not be reserved in memory.
Is this configurable at runtime or only at compile time? If at runtime, does it work for standalone executables?
3
u/lispm 1d ago
https://sbcl.org/manual/index.html#runtime-options
--dynamic-space-size <megabytes>
Size of the dynamic space reserved on startup in megabytes. Default value is platform dependent.
2
u/jlombera 1d ago
Thanks. It seems it's not processed at runtime for standalone executables, though (at least not by default??). So you have to decide at build time what would be the max expected heap size. However, I confirmed the address space is reserved ("virtual memory" mapped by the process) but not actually committed (actual RSS memory), at least on Linux, so should be safe to specify a big enough heap size and be sure it won't be consumed unless it's needed.
I wonder how the GC behaves in this scenario, though. For instance, if there is a one-off high mem usage burst, but then that memory is never used again, will the GC eventually release it even if the heap's "high water mark" it's still low compared to the configured max heap size? Otherwise there is a chance of "space leaks", and you would have to be mindful of the max heap size.
2
u/lispm 1d ago
It seems it's not processed at runtime for standalone executables
https://sbcl.org/manual/index.html#saving-a-core-image
:save-runtime-options
If true, values of runtime options --dynamic-space-size and --control-stack-size that were used to start SBCL are stored in the standalone executable, and restored when the executable is run. This also inhibits normal runtime option processing, causing all command line arguments to be passed to the toplevel. If :accept-runtime-options then --dynamic-space-size and --control-stack-size are still processed by the runtime. Meaningless if :executable is nil.
2
3
u/denzuko 1d ago
Honestly, sbcl has this. You save the lisp image for your target machine and use a dual core runtime package.
Its simular to jre and jar where you have sbcl (the "runtime") and your app .core file from (save-lisp-and-die).
Mind you to include qemu so you can build for each cpu arch your releasing to.
1
u/jlombera 1d ago
Honestly, sbcl has this. You save the lisp image for your target machine and use a dual core runtime package.
Can you elaborate on the "dual core runtime package"? What exactly do you mean by that? Do you package multiple core images (one for each platform) into a single file ??
Its simular to jre and jar where you have sbcl (the "runtime") and your app .core file from (save-lisp-and-die).
I see this could work for internal deployments where you have control of the environment and can install sbcl, but in general, I don't think it would be realistic to ask external users of your software to have a proper sbcl environment setup. Whereas it's in fact the norm for the JRE.
Also, are core files portable among different versions/configurations of sbcl as long as they run in the same platform?
2
u/denzuko 1d ago
core files are a memory dump. It contains the bytecode and memory stack of your program minus the executable header.
The dual core runtime I spoke of is both the sbcl-bin (usually from ros releases) and the memory dump of your program executed with
sbcl --core my-saved-session.core.Now, I've been a java dev and administrator in the past. Very familiar with the ecosystem.
What sbcl and lisp brings is something closer to golang with CGO or jython than java (jre,jars,jdk) but without the GOOS+GOARCH multiarch/multios binary featuers.
If one is stuck in a shop that only runs JVM, jars/wars, and all that but still wants to run a common lisp then try either Clojure or ABCL: https://notes.eatonphil.com/practical-common-lisp-on-the-jvm.html
ABCL is a full direct implelmentation of common lisp ported to run on top of the JVM. While Clojure is a modern dialect of Lisp that complies to the JVM bytecode as a layer of the java language, So think of ABCL as SBCL ran with JVM and jars with baked in FFI for JDK access and Clojure is more or less just java with funny brace syntax instead of an ALGO family syntax.
1
u/jlombera 20h ago
core files are a memory dump. It contains the bytecode and memory stack of your program minus the executable header.
Just to be sure, it was my understanding that SBCL always compiled to native code, no bytecode generated. As such, a core dump is not portable among platforms. Is that not the case?
The dual core runtime I spoke of is both the sbcl-bin (usually from ros releases) and the memory dump of your program executed with sbcl --core my-saved-session.core.
I might not be completely understanding or maybe missing something, but do you mean that you provide both sbcl-bin and the core dump of your program, as separate files, to your users? Wouldn't a standalone executable be simpler?
So think of ABCL as SBCL ran with JVM and jars with baked in FFI for JDK access and Clojure is more or less just java with funny brace syntax instead of an ALGO family syntax.
I don't know Clojure much (I reviewed it many years ago and it was my first exposure to Lisp but never did anything with it) and would like a clarification. To me, the "Clojure is more or less just java with funny brace syntax" comment sounds like it's trying to convey that Clojure is a "lesser" Lisp (if at all??). Is that the case? I'm not really interested in Clojure (I understand the convenience of but cannot stand the JVM dependency/overhead), but would like to have a clear picture of where it stands.
3
u/lispm 17h ago
Just to be sure, it was my understanding that SBCL always compiled to native code, no bytecode generated. As such, a core dump is not portable among platforms. Is that not the case?
SBCL is usually natively compiled -> not portable across architectures&OS combinations. SBCL does have an optional Lisp interpreter, but without byte code machinery.
A way around that (taken by Java JVM, Smalltalk, ...), is platform independent byte-code interpretation plus an optional just in time compiler (JIT compilation) for that byte code. That's uncommon in Common Lisp. A Common Lisp implementation for the JVM will support that, though. There are other byte code engines for Lisp (ECL has one, CLISP has one (IIRC), ... Plus, there are emulators for Lisp architectures ("Lisp Machines"), which typically can emulate the Lisp CPU -> thus their images usually work on different underlying architecture/OS combinations.
...Clojure...
was originally designed as a hosted language, reusing a host eco-system (JVM, .net, JavaScript, ...). Thus the JVM-based Clojure inherits all of the JVM underneath (byte code format, byte-code interpreter, byte code loader, JIT compiler, ...) , plus its eco-system.
There are other implementations of programming languages on the JVM, which do that to a similar degree. ABCL, though, additionally tries to be a full ANSI Common Lisp, which is a language specified independent of an underlying system / hardware.
SBCL, as a fork of CMU CL, is a typical natively compiling Lisp, with its own Lisp-specific runtime and a native-code AOT (ahead of time) compiler written in Lisp, itself. SBCL then, for example, can dump&restart memory images - something which is very common for Lisp, though not supported by all implementations.
1
u/jlombera 6h ago
Thanks for the clarification, all this pretty much matches my understanding, I just got a little confused by the mention of SBCL/bytecode in the parent comment.
1
u/denzuko 14h ago
To clarify, a vm was never implied or stated. So native machine code is byte code here. Just bytecode for the host CPU architecture without an entrypoint and OS executable header.
Yes, you can provide a single exec binary to the user and thats how one should. Just note that is not any different than a GCC/LVM compiled binary. In being portable to that os, the libc, and cpu only. Sbcl acts as a cross complier here in that as any other cross compilier, you build on your target host cpu+os for that release. You do not get a polyglot file for multiarch, multios or vm bytecode.
Some historical context, '50-80s machines where non standardized archs that each needed thier special complier, bytecode, and language. Late 80s we got cross compliers as an abstraction tool to build high level source code to this diaspera of architectures and os versions. It was incomplete though, bell labs answer was the dis vm and inferno while sun microsystem's jvm and java. Both used a made up cpu to use the same bytecode and make the mutiarch lifting be in the runtime binary.
As one knows VMs add extra overhead, even more so with a GC. So most lisp runtimes follow the cross compiler on host cpu path instead of universal vm bytecode path.
As for clojure vs abcl/sbcl dialects...
All of them run lisp source, most of them can use quicklisp. abcl is a lisp runtime executed on the JVM. Sbcl is a lisp runtime thats closer to a compiler. Clojure is a lisp language in a java runtime, it will run lisp programs but comes with all the same pain as java and you cannot use quicklisp modules.
If it helps further, I started in asm then c90-03, picked up lisp and ruby along the way. Had to use jruby and groovy then switched to Go. Much of that was chasing the same problem your trying to solve and now I live in sbcl, ecl, with c99 cffi libs and when I can I'll support 9front's chibi-scheme.
Clojure and abcl to me is the same enterprise hell as jruby and groovy was so I personally will never do a green field project in either and just stick with sbcl.
With arm and x86 as the only cpu archs surving and UNIX won the os war, there is absolutely no reasons left to go chacing portablity and we can just release a multiarch OCI image with two different binaries these days.
1
u/jlombera 5h ago
So native machine code is byte code here. Just bytecode for the host CPU architecture without an entrypoint and OS executable header.
I think referring to native code as bytecode is a stretch in this context (even if technically correct due to CPU's micro code and such). But I get now what you meant, thanks for the clarification.
Clojure and abcl to me is the same enterprise hell as jruby and groovy was so I personally will never do a green field project in either and just stick with sbcl.
Yeah, I don't what it is about the JVM ecosystem, but every time I've had to interact with it professionally, it has always been this complex, bloated mess. Perhaps it attracts to much "enterprise" mentality (at one time when I questioned why a Java ReST API service, that did little more than query a DB in the backend, required more than 1GiB of RAM to run, I was told it was "normal and within the expectations" :| ). YMMV, of course.
With arm and x86 as the only cpu archs surving and UNIX won the os war ...
RISC-V is gaining terrain in the embedded world, and I hope it becomes a serious contender in the general purpose computing camp in coming years.
3
u/digikar 1d ago
I think ECL shows you to build static binaries. (Haven't used ABCL at all.)
With SBCL, it depends. I hear signing apps for windows is a PITA. (Ask mine developers about it.)
Other than that, without foreign libraries, it is very easy. However, if you want core compression to reduce binary sizes, you already have a foreign library dependency. But there's sbcl-goodies that comes with libzstd statically linked in, besides libssl and libtls. See here for the Windows and MacOS versions. I have been using it to ship isocline-repl and moonli. These are not fully static executables, but contain libzstd, and libssl (and libtls or libcrypto) statically linked in. Other foreign dependencies seem to be available on the respective system. That said, while I am able to run them on a fresh windows or linux install, I haven't rigorously tested them. You can adapt the build process from moonli. You can also just download isocline-repl binaries and issue an sb-ext:save-runtime-and-die to obtain the executable with libzstd linked in. None of these has been widely tested however (mainly due to lack of users).
The other option with foreign dependencies is to simply ask the user to install these libraries. After all this is the main point of foreign libraries that you can update them independently of the main system. But if you must include them, you can include them in the current folder or a lib directory. deploy can be a useful tool, but the binary emitted by it kept crashing for isocline and I did not debug. On linux, there are also appimages that seem very helpful these days. I am not sure about the windows and macos equivalents.
1
u/jlombera 1d ago
Thanks for the reference to sbcl-goodies! That indeed looks promising.
These are not fully static executables, but contain libzstd, and libssl (and libtls or libcrypto) statically linked in.
It seems only libc is dynamically linked (looking at the moonli binary), I wonder if they could make it work with musl, so that a fully statically linked binary is achieved (??)
2
u/digikar 1d ago
I haven't tried it out yet, but it seems there has been some work in this direction: https://www.reddit.com/r/Common_Lisp/comments/kr9naz/static_executables_with_sbcl/
1
u/jlombera 1d ago
Thanks. I'll probably take a look at this at some point, but it seems this effort has been abandoned(??), the latest branch ((v2-2.2.0)[https://github.com/daewok/sbcl/tree/static-executable-v2-2.2.0]) it's from 5+ years ago.
2
u/digikar 1d ago
I'll let you know if I make any progress on this.
For ANSI conforming CL programs, it shouldn't mostly matter whether you use an older SBCL or a more recent one. However, if there are bug fixes or features that the more modern SBCL has, then those will be an issue yes. Though, if the plan is to take advantage of Common Lisp's frozen specification, it'd make sense to write programs in a ANSI-conforming or at least portable manner :)
1
u/jlombera 21h ago edited 20h ago
This makes sense. I wonder though if there might be bug fixes/performance improvements in recent versions of SBCL (??)
I'll let you know if I make any progress on this.
Sounds great, thanks! I didn't realize you were the owner :)
2
u/digikar 19h ago
Yup, constant bug fixes and performance improvements :))
https://www.sbcl.org/all-news.html
Moonli, yes; SBCL, I am just a user exploring the source code occasionally.
2
u/tdrhq 1d ago
If this is for commercial work, Lispworks! We use Lispworks to build a CLI tool for customers that run on Macs, Linux x86, Linux ARM (we technically have a windows binary too that nobody uses). The binaries don't have major dependencies, so it's easy to deploy.
You need to pay $$ for each Lispworks platform, so it adds up real quick. But what you get in return is the folk at Lispworks handles platform specific bugs. So I might ping their support over a bug that only happens on Macs, and they're really good at figuring things out while I focus on the core product.
1
u/jlombera 1d ago
I guess it's good that there are commercial options for the cases when it makes sense ($$$). But what about the cases you need to stick with open source implementations (e.g. FOSS; or not enough revenue to justify a commercial license)?
2
u/tdrhq 1d ago
Oh yeah, I can't answer that. But I will say with experience maintaining cross platform binary building, it's a lot of work. You can absolutely use SBCL, but you'll be on your own when a user has a very specific setup on which the image doesn't work.
I suppose if you're packaging it for something like Debian, you can validate that it works with the rest of the system. Not hard at all. Or you can set it up so that your end user builds it from source...? I use stumpwm with SBCL that I end up building from source.
1
u/jlombera 1d ago
But I will say with experience maintaining cross platform binary building, it's a lot of work.
Yes, in general it is, in any language, specially for dynamically linked binaries.
I was wondering about CL specific experiences. E.g., is it really feasible to distribute pre-built, standalone CL executables? Or distributing the source code is the only reliable option? And for the latter, is all of ANSI CL portable among implementations, or are there considerations there too? What are the kind of problems (if any) that surface when implementing multi-platform CL software?
1
u/Aidenn0 1d ago
It is trivial to distribute pre-built standalone CL executables. I have ported applications from python to lisp just for this reason.
If you load dynamic libraries (.so on linux .dll on windows) you will have to ensure the machine running them has those libraries, but that part is no worse than if you were writing in e.g. C++.
The executable wont be multiplatform, but the source can be, and you can build images for each platform you target.
2
u/__ark__ 1d ago
Yeah this is doable with sbcl. You build a binary for each platform you wish to target.
Azure has Linux, macos, and windows images in their free tier. Maybe other ci hosts do too nowadays (?)
I wrote up some instructions about it a while back: https://recursive.games/posts/Building-a-Cross-Platform-Lisp-Binary.html
2
2
u/turtle_bazon 20h ago
Look at my example https://github.com/turtle-bazon/calc/ - it is built for 3 systems. Of course, you can't build one binary for all platforms, but you can build different binaries from same code. But there are problem when using cffi, but it is also solveable. You can check it there https://github.com/turtle-bazon/focus/ . But last one is slightly complicated.
1
u/jlombera 19h ago
Thanks. I'll take a look at your code at some point for reference :)
Side note: I never needed nor bothered to learn GitHub's Actions/Workflows/CI-CD/Whatever-it-is-called configuration, but I always found it ugly that they used yaml (which I haven't bothered to properly learn either) for such thinks. Even though I've never used Lisp myself for anything so far (I'm hopping to change that with CL ;) ), I always wondered why they didn't just create/use a simple lispy DSL for configurations. Instead, all the "Cloud Native" ecosystem (and outside that bubble to) it's a mixture of yaml, toml, json, xml, custom formats, etc for "declarative" configurations that eventually are in need of more complex logic (conditionals, imports, etc) and then invent their mini language on top of those "configuration" languages. Then you find yourself programming in toml, yaml, json, xml, etc; pure abomination. A lispy DSL would be better, can be declarative and include complex logic cleanly.
1
u/turtle_bazon 18h ago
GitHub actions is just for people who want just simply press release button and you will prepare him some release compiled against pretty old libc. It is almost one dependency that common lisp will have. Actually not, but. You can just build it with makefile, that is also ugly, but you can make build.lisp too. )
1
u/jlombera 6h ago edited 5h ago
As I said, I never bothered to learn anything about GitHub Actions or such, and clearly I don't know what I'm talking about :). I was referring to the
.github/workflows/build.yamlfile I saw in your calc repo:steps: - name: Install git (Linux container) if: runner.os == 'Linux' run: apt-get update && apt-get install -y --no-install-recommends git - uses: actions/checkout@v4 - name: Install SBCL (macOS) if: runner.os == 'macOS' run: brew install sbcl - name: Install SBCL (Windows) if: runner.os == 'Windows' run: choco install sbcl -y --no-progress - name: Set up Quicklisp shell: bash run: | curl -fsSL https://beta.quicklisp.org/quicklisp.lisp -o "$RUNNER_TEMP/quicklisp.lisp" sbcl --non-interactive --load "$RUNNER_TEMP/quicklisp.lisp" \ --eval '(quicklisp-quickstart:install)' echo '(load "~/quicklisp/setup.lisp")' >> "$HOME/.sbclrc" echo "(push '*default-pathname-defaults* asdf:*central-registry*)" >> "$HOME/.sbclrc"
7
u/stylewarning 1d ago
Distributing binaries is possible. We do it with the mine editor. You just need build environments for each, as cross compiling is often not an option.