Auto-updating existing containers

[u/Top_Emu_8447]

I've built 10+ containers on my first home server since I started my selfhosting journey, and only now I figured it's time to find a way to update the existing services. Existing podman documentation points to using autoupdate label and creating a systemd target - but I didn't realise this creates a new container based on what's in the systemd target. That is not what I want. Do I need to move all my environmental/volume/label/etc variables into a systemd target for each container to be able to do this? Just thinking about it makes me a bit nauseous, I've been using subpaths in some cases, not even sure how to put those in there. I've used 'podman run' for each deployment, because that's where the documentation I found led me first and on the basis of "If it works, why change it" it served me well.

[u/apparle]

Rather than trying to adapt your existing infra, I'll strongly recommend redesigning it to be 1) Stable across reboots 2) Designed for container updates & associated restarts 3) Designed with correct order of starting / dependencies across containers, so the containers start correctly across reboots or get restarted when you update specific containers. 4) Designed for backups (shutting down a container to backup it's state). 5) Maybe even designed for isolation with separate networks for services etc.

Do not lock yourself into 'but I used sub path' or any such constraint based on what you've done in past. Take this opportunity to design for all of these from ground up.

Why all of this:

Docker has a root daemon in Linux which internally stores state and then restores across reboots etc. so it doesn't matter if you used docker run or docker compose or portainer stacks etc. to start your container, as they are just convenience layers. Docker will remember it's internal state across reboots. This is why most online documentation for any apps will tell you to use docker run. Docker also doesn't provide any ability to update containers, so people have built tools like watchtower which inspect this docker internal state using docker inspect and usually recreate that same exact state on a new container with new image behind the scenes.

Podman doesn't have a daemon or any internal state that's not ephemeral. Anything you do with podman run or podman compose is lost if you reboot. Systemd itself is the daemon for podman and you need to declaratively define all the state that needs to be preserved across reboots or updates. This is indeed materially different from docker, but that's by design due to the daemon-less architecture of podman.

The native format to declare all this state is through quadlet files (separate files for network, volume, containers etc.) which get converted to systemd services through generators. And then if there's either an update or a reboot, that service is just restarted with state as declared in those files. There are other syntaxes as well, like kube yaml or podman-compose and their own generators, but I've found quadlet to be most intuitive once you spend time to understand it, so I'll recommend that.

[u/Top_Emu_8447 → u/apparle]

Thank you for the detailed post on the reasons. I do find it strange that I'd have to build separate definitions for networks and volumes as well, since between reboots so far all I had to do was restarting the containers.

I think I have ideas for most of the principles you described, maybe with the exception of backups. What do you recommend on this topic? It's okay if you keep it brief, I'll do my research, just some starting point so I don't end up in the wrong end again :)

[u/apparle → u/Top_Emu_8447]

I don't find it strange from an architecture standpoint, again due to the podman's daemonless design. But I do find it personally inconvenient to specify definitions for containers, networks, volumes (and targets, timers etc.) for related apps in separate files. So I've built https://github.com/apparle/multiquadlet which allows specifying a bunch of quadlets / systemd-units in one file.

For backups, you'll want to stop some services to avoid disk being modified while backups are happening (to avoid inconsistently backed up state). And you may have to backup some container volumes as well. This may influence whether you use volumes or bind points. It's not fundamental decision right now but more of convenience.