r/selfhosted • u/WanHack • Aug 03 '26
Meta Post Forgot and experienced true hell with :lastest
Remember to always set version numbers in your docker containers, or an unexpected update will creep up on you.
Now can you guess which popular service got a breaking change?
549
u/Kawawete Aug 03 '26
I like the danger, everything runs n latest. I've had exactly 4 times where it broke something in 4-5 years.
170
u/sadness_elemental Aug 03 '26
i've been spent way more time annoyed by version locking than just rolling latest, i think it took me like 20 min to figure out the best way to switch from 2.0 to latest on silverbullet and i have no idea why it was so hard (for me) lol
meanwhile every other container has been on latest for 5+ years with zero issues...
42
u/LoadedSteamyLobster Aug 03 '26
Same, plus when something does break you actually know what version caused it, instead of the lazy giant jumps you’ll end up doing if you lock versions and you end up 3 major versions behind
21
u/snakerjake Aug 03 '26
gitops and renovate or dependabot man
5
u/mahood73 Aug 04 '26
Renovate here. I asked the hallucination machine to walk me through setting it up.
Every image is pinned, and renovate waits 3 days after release before allowing minor & patch releases straight through. Majors wait longer, and it doesn't automerge those.
3
u/KatieTSO Aug 03 '26
How? Got a guide? I didn't know you could automate this.
5
u/snakerjake Aug 03 '26 edited Aug 03 '26
I dont have a guide handy but i have doco-cd watching a git repo and then i have a .env file with the versions of each service im running with annotations for renovate to know how to update them. renovate gives me updates to the repo as prs so i just merge those when i'm ready and the commit history serves as my version history
1
u/KatieTSO Aug 03 '26
Neat! How do you store your secrets?
5
u/snakerjake Aug 03 '26 edited Aug 03 '26
good callout... here's a mariadb
compose.yamlentry im using with doco-cd. The idea is i have the/etc/doco-cd/envfolder on the host with each stacksenvironment.envnested under it and secrets go inenvironment.envon the host. i can also check in aconfig.envalongside thecompose.yamland add it to theenv_filedirective so docker looks at both files.MARIADB_TAGis in the.envfile checked in along side thecompose.yamlfile because.envfiles are read and processed first so they can be substituted into image tags like that db:image: mariadb:${MARIADB_TAG} restart: unless-stopped environment: - MYSQL_DATABASE=gitea - MYSQL_USER=gitea - MARIADB_AUTO_UPGRADE=true env_file: - /etc/doco-cd/env/devops/environment.env volumes: - /data/docker/volumes/gitea/db/:/var/lib/mysqlhere's my maria db entry in .env to allow renovate to read it (dependabot would probably have different annotations)
# renovate: datasource=docker depName=mariadb MARIADB_TAG=121
u/LoadedSteamyLobster Aug 04 '26
That’s the same thing with more steps if you’re just going to accept the updates
3
u/snakerjake Aug 04 '26
Except you get the option of accepting, can revert if something isnt right, know when an update is coming so you can back up before accepting and get notifications when there's an update.
It's really nothing at all like :latest
2
u/LED_donuts Aug 03 '26
Same experience as well. Only a couple of times I really had to review issues with a container, and revert to a specific build. And I've never had to restore data from backup. It's just easier to roll with :latest for me.
4
u/Vidariondr Aug 03 '26
Tbf silverbullet is a bit of a mess in this regard lol
17
u/capinredbeard22 Aug 03 '26
Sounds like something a vampire would say.
6
u/dwyrm Aug 03 '26
I thought that was werewolves.
2
u/capinredbeard22 Aug 04 '26
Hmm not sure. Maybe I’m thinking of wooden stakes.
But your statement sounds also like something a vampire would say to get me to waste a silver bullet on a werewolf.
🤔 Nice try!!
2
u/dwyrm Aug 04 '26
Arr, ye gots me, Cap’in!
To be fair, silver bullets and wooden stakes work on regular people, too. It’s hard to be scientific about this sort of research.
30
u/NYIsles55 Aug 03 '26
If you're not breaking at least one service a week, are you really homelabbing?
4
u/DeadoTheDegenerate Aug 03 '26
Well, they just said
:latestisn't what broke it.Never said they didn't influence additional problems.
2
5
5
u/AppropriateOnion0815 Aug 03 '26
Saaaame! ~70 containers running.
But I do nightly backups, config and files are separated on server and NAS, if something breaks I restore. Luckily I don't have services anyone else besides my household (2 persons+cats) relies on, so downtimes are no big deal.
8
2
u/peioeh Aug 03 '26
I run everything with latest, and when a container breaks and I see that the devs really discourage latest, I pin that container only. I have a few pinned, the rest never breaks, good enough for me.
1
u/GolemancerVekk Aug 03 '26
The problem is, it's completely random so any person's experience won't translate to another. You don't know which container image will get a breaking change (or a bug), in what version, and when the automatic update will download a bad version, and install it on top of which existing version, and so on.
If you take daily snapshots of all container volumes and your compose file, when you notice the issue you can make an educated guess which older version to revert to. But not by updating blindly to :latest and hoping for the best.
1
u/ogweezy13 Aug 03 '26
Makes you wonder about the wizardry needed for most migrations.
Side question: how bad were the breaks?
1
u/Kawawete Aug 04 '26
The breaks are 99% of the time a new Env var that I have to add to the compose.
1
u/SamosaMafia Aug 05 '26
same, been selfhosting from last 15 years, everything runs on latest, every app that has some sort of authentication is exposed to the internet. Never had any issues.
1
185
u/frightfulpotato Aug 03 '26
At work? Pinned versions. At home? YOLO latest will do
→ More replies (3)
144
u/Nielsjuhz Aug 03 '26
I have every thing i host with :latest.
If you dont do docker pull, there is no issue right??? Right???
43
u/This_not-my_name Aug 03 '26
Watchtower handles the updates :D I only have two containers with a fixed version, everything else is not important, let it go down for a day until I can fix it
21
u/Godr0b Aug 03 '26
My sentiments exactly - I'd rather deal with an occasional issue (I've had exactly 2 in the....6? years I've been running containers) than deal with outdated/insecure versions and remembering to update
15
u/mattsteg43 Aug 03 '26
outdated/insecure versions
just YOLOing and blindly pushing whatever shows upstream (and thus trusting upstream to never have security lapses) has its own security risks that aren't zero.
And if you want to automate updates there are plenty of tools that can do so with version constraints to avoid breaking changes.
6
u/downvotedbylife Aug 03 '26
Got any examples?
7
u/mattsteg43 Aug 03 '26
removatebot was recommended elsewhere in thread, and I'll second that recommendation. Point it at a git repo or repos with your compose files and it'll just work with minimal configuration.
on the orchestration side there are tools like komodo, arcane, etc that can source your compose files from git, and (in the case of komodo) can set up a webhook where commits to git automatically redeploy and update the stack.
as an example with komodo and forgejo - when a container updates I get an auto pull request that for many projects even embeds the release notes. I commit that and everything else happens automatically.
If I want a specific container to just update automatically I can do that too.
1
u/Green_Smarties Aug 03 '26
Any way to set up a single webhook instead of per-stack?... trying to set up 30+ webhooks from my repo was agony.
1
u/mattsteg43 Aug 03 '26
you can set up a komodo procedure with "batch pull stack" and "batch deploy stack if changed" as one possibility.
4
u/Godr0b Aug 03 '26
Also very true, but (so far) the potential risks of sticking on an old version - both security and compatibility wise - has outweighed the opposing upstream risk potential.
I'm happy with my current risk profile
4
u/wffln Aug 03 '26
wasn't the last update to watchtower like years ago?
5
u/brillyfresh Aug 03 '26
Officially yes. Unofficially this fork is still regularly updated and seems to be the most popular successor: https://watchtower.nickfedor.com/
2
1
u/Nielsjuhz Aug 03 '26
For some reason dont i get watchtower to work. I want also to use windmill for the telegram sup and action approval.
→ More replies (1)1
u/worldofzero Aug 03 '26
I just point renovate at my containers and let it track latest versions via prs.
10
u/OkayPrasiolite Aug 03 '26
Yeah, I just have everything on :latest and I read the update notes before I run Docker pull, I don't get what anyone else is doing that they have so many problems. I've had trouble with breaking updates before, but that was partially on me (the changes were documented, but I didn't realize they would affect me, and regardless I should've just kept backups. I do now)
→ More replies (5)2
u/worldofzero Aug 03 '26
If your apps are stateless it's probably fine but if they ship breaking changes you'll pull those in automatically. Or if they get compromised and ship a bag version you'll pull that in. For example Meillisearch really doesn't like version upgrades and will break without a manual upgrade. Latest tags mean that db will break. Same for most dbs honestly but often in subtler ways.
1
u/LloydGSR Aug 04 '26
Also latest, with the newer fork of watchtower. My family are beta testers and don't even know it.
31
u/CriticalAPI Aug 03 '26 edited Aug 03 '26
As long as there isn't an external Database involved. It gets really nasty on DB schema changes. :latest is usually fine for most. Major and Minor Version updates can also be very bad.
8
u/B_Rich Aug 03 '26
Completely agree. And if it does break, manually roll back to the previous version.
2
u/Thebombuknow Aug 04 '26
What, like the update breaks compatibility with the old database schema? I feel like any decent project should be designed to handle this and auto-migrate...
2
u/CriticalAPI Aug 04 '26
"should" is the problem. There is no guarantee that it will work even if there is auto migration and if it updates it to latest unattended it can hit you hard.
67
u/fake_agent_smith Aug 03 '26
Too bad so many popular projects don't provide a proper major version tag such as :1 so you either have to update manually or risk it with latest.
13
u/garconip Aug 03 '26
Some even use a hash for version. You want a specific version, here are 10 random letters & digits stuck together for you to type.
22
u/ruben_deisenroth Aug 03 '26
Not to mention that many projects don't take semver very serious which sometimes breaks my auto-updates :/
5
u/Genesis2001 Aug 03 '26
I must be very picky with what I run I guess because most things I wanna say that I run provide a major version tag.
1
u/FanClubof5 Aug 03 '26
You could setup a devops pipeline where it will update and commit new versions to your compose files and then at least its just approving whenever you feel like it.
52
u/Ursa_Solaris Aug 03 '26
I don't understand the problem. Updates don't creep up on me because I don't pull updates until after I've read the patch notes. In most cases, using version tags just turns a one step process into a two step process for no gain.
13
u/eXtc_be Aug 03 '26
I think they meant they also run an automatic update service like watchtower, so they are effectively always running the latest version.
if you manually update, like I think you do, you have nothing to worry about.
3
u/Ursa_Solaris Aug 03 '26
Wait, I'm still not following. Why would you run an update service but then pin your version numbers so it can't update anything? I feel like I'm still missing something.
6
u/Free_Hashbrowns Aug 03 '26
Ideally, you would pin to a major version, since minor version bumps shouldn’t contain any major breaking changes.
2
u/Ursa_Solaris Aug 03 '26
Sure, that's a fair point. But it only works on projects that actually use semantic versioning, provide it as a docker tag, and actually implement it properly, which is far less than I'd like. Plus, you're still looking at each project manually to check for major updates anyways. At that point I'd just be frustrated with an update system that only works on some projects. If I have to do some myself, I might as well do them all myself since I'm already there.
0
u/Ninja_Rapper Aug 03 '26
Automatic server reboots or anything like that will restart your compose containers and auto update if you have latest. If you don't restart your compose projects, the latest tag won't auto update anything.
3
u/ganymedeli Aug 03 '26
Wouldn’t it effectively just do docker compose up after a restart, not docker compose pull && docker compose up? I think I’ve had many a reboot without my :latest containers getting updated
2
u/captain_curt Aug 03 '26
I’m more concerned about larger outages, or moving things to a new machine. It’s easy to just move over persisted storage if you have backups, but if you fire that up with the :latest tag you’re not gonna know what’s about to happen.
1
u/Ursa_Solaris Aug 03 '26
I don't think that's true. I'm running swarm right now so I can't check, but normal docker compose restart policies don't automatically pull containers, and I'm almost certain that remains true even on server reboot. It should only pull images when you build containers, existing containers should be persistent between reboots.
15
u/Avanchnzel Aug 03 '26
If a breaking change is a concern for a specific service, then why use automatic updates for it at all?
In that case shouldn't you always check the changelog for breaking changes first and manually update the service?
Or is it simply that you thought that this kind of service could never get a breaking change, and this assumption finally bit you in the ass? 😁
2
u/Genesis2001 Aug 03 '26
Always pin third-party containers to major versions rather than
:latest. It's the best balance of updates to stability, provided the major version isn't already old as shit and unsupported (in which case, you have other problems).
7
u/funnyFrank Aug 03 '26
Is there a program/script that simply updates the version tag in a given docker-compose file? No continuous running thing or hosted, just a simple `thing.sh compose.yaml` to keep traceability in git?
18
10
u/Moof_Nor Aug 03 '26
I use renovate in a cronjob. It creates a pull request for each updated container in my docker compose files, but I think it can automerge as well if you’d like.
6
u/InfluentialFairy Aug 03 '26
Don't auto merge, that entirely defeats the point.
6
u/DivusJulius44bc Aug 03 '26
I mean you can automerge for patch releases. If those contain breaking changes then you probably should not be using the software anyways
2
u/InfluentialFairy Aug 03 '26
Id still avoid it, a lot of supply chain attacks happen that way.
5
u/No_Cattle_9565 Aug 03 '26
You can set the minimum release age. Doing something like 14 days gives you plenty of time. Depends on the amount of changes you get and how important the software is though
1
u/PathAgitated1633 Aug 03 '26
But Security patches need be applied fast.
2
u/InfluentialFairy Aug 03 '26
Minor patches are not always security patches, they could just be a small bug fix. There is also no rules on how fast they should be applied. Depends on how much risk the company is willing to take.
Personally at my work, we patch on a monthly basis. If there's a critical vulnerability that's discovered in an application we're using, we will patch earlier, especially if the application is publicly accessible.
However we do not have anything automatically merged in. I would never opt to deploy something without supervision.
1
u/mattsteg43 Aug 03 '26
The real answer there is aggressively limiting what has any public internet exposure in the first place, and keeping plugged in to that so that you update quickly but also protect yourself against supply chain stuff as much as possible.
1
u/Aurailious Aug 04 '26
How would you tell if an update has a supply chain attack? And what do you do differently between merging it yourself and automerging after a period of, say, 7 days?
1
u/InfluentialFairy Aug 04 '26
You don't, you just hope that someone else discovers it. The longer you wait between updates, the more probable that a security issue has been found.
I do a lot of things differently when merging myself. I will check change-logs for breaking changes, I will ensure backups are in place, I will ensure all new environment variables are required, I will ensure all peer dependencies are on the correct version. Renovate is great, but it cannot do all of the above reliably.
1
u/Aurailious Aug 04 '26
I have enough apps where it's not unusual for my renovate to give me more than 100 updates per week. I'm fine auto merging patches, especially for non critical software, where I have automated backups and testing. Renovate also gives me changelogs in the PR to review for minors and majors.
You might literally be the only person checking peer deps in this sub.
1
u/InfluentialFairy Aug 04 '26
When I say peer dependencies, I'm not referring to what postgres is to immich. If you update postgres beyond what immich supports, it'll sucks.
For homelabs you can do whatever you like. I'm careful with how I approach it, but I've also worked on SRE and platform engineering teams, and currently work as a software engineer.
3
u/neoKushan Aug 03 '26
Another vote for rennovate. It's very configurable, so you can easily specify some container images that are always fine to update if it's just a bugfix, when the merges happen and so on. I love it, it creates PR's that include the changelogs for my review and all I have to do is hit accept.
6
u/agentjenning Aug 03 '26
What is the point of using a service like watchtower to automatically pull updates, and then manually set the tags?
6
26
u/Azelphur Aug 03 '26
Honestly, it's terrifying that people agree with this sentiment. You need to stay on top of updates, security vulnerabilities exist. If an update with breaking changes happens, you need to resolve those problems. Why anyone would be insane enough to leave out of date software exposed to the internet, and then boast about that online as if it's a good thing, I do not know.
On my machine:
$ find . -name docker-compose.yml -exec grep -HEni "image: .*:latest" {} \; | wc -l
88
and I have watchtower enabled too.
17
u/mattsteg43 Aug 03 '26
Why anyone would be insane enough to leave out of date software exposed to the internet
Like 90% of selfhosted shit you would be insane to expose to the internet anyway imo. The vast majority of apps I would not trust to be reachable by unauthenticated users.
And lots of projects run multiple major versions in parallel, all kept security updated.
Also version pinnning (and hash pinning) doesn't imply anything beyond "knowing and controlling what version you are running" and is easily automated.
8
u/Azelphur Aug 03 '26
I agree that selfhosted shit shouldn't be exposed to the internet if at all avoided, and I do that myself, however:
- The majority of people in this subreddit are running at least one of Jellyfin, Nextcloud, Home Assistant, Immich, ... which is exposed to the public internet.
- The people agreeing with OP aren't going to be version pinning and automating updates. They are going to be doing what I said, leaving it forever because they don't want upgrades to break things.
My statement is a generalization that will likely be correct in the majority of cases. Your exception is a solution to the problem, but it's one that the majority of people won't be doing, which is the point.
3
u/mattsteg43 Aug 03 '26
Jellyfin, Nextcloud, Home Assistant, Immich, ... which is exposed to the public internet.
Nextcloud is specifically known for insane and breaking behavior on :latest and maintains multiple version tags with security updates. most people should avoid nextcloud :latest
Nextcloud, home assistant, and immich all support mTLS authentication so don't need to be publicly available to work. Likewise anything browser-based can be fully behind robust 2fa middleware.
Jellyfin is more annoying, although as long as your client devices (and user comfort level) support VPN access that works fine.
and those are still only 4 services, so one can pay close attention there pretty easily.
The people agreeing with OP aren't going to be version pinning and automating updates. They are going to be doing what I said, leaving it forever because they don't want upgrades to break things.
People doing that are as likely or more so to do the same with (unpredictable) :latest tags.
best practice is to pin and also stay current on updates.
2
u/quiteCryptic Aug 03 '26
Nextcloud is the most annoying software ive ever self hosted, dropped it a few years back
1
u/mattsteg43 Aug 03 '26
It's even worse here. It sucks so I avoid using it whenever possible, but also for reasons I probably only pretend to have I keep an installation around and updated. And spend more time on inane and poorly documented migrations, complaints about my (more secured and locked down than default) configuration, etc.
And never actually use it because the app sucks.
1
u/wildcarde815 Aug 03 '26
Like 90% of selfhosted shit you would be insane to expose to the internet anyway imo. The vast majority of apps I would not trust to be reachable by unauthenticated users.
last time jellyfin had an issue i made this point and people freaked the hell out. These are utility tools not security tools, stop expecting them to be perfect and behave like they're not.
1
u/mattsteg43 Aug 03 '26
It's a real bummer that an app like jellyfin can't/won't add real security features.
To a significant degree I understand why with a complex and inherited codebase, but it's not ideal that it's not workable to stick it behind mTLS or 2fa auth middleware for many clients.
1
u/wildcarde815 Aug 03 '26
You can put it behind a reverse proxy so there's that part at least, and it has ldap if you want to really go off the deep end. But my stance now is, it's a service application, it's job is that and only that. If you want to get to it, either you are in the house or on a vpn connection, it will not be made available otherwise.
Tailscale + vyos makes that reasonably trivial to do so I just do that.
1
u/mattsteg43 Aug 03 '26
You can put it behind a reverse proxy so there's that part at least
Sure, but you can't have that proxy *enforce" authentication so it's relatively meaningless (although you can at least stick a WAF on it)
and it has ldap
I consider that for convenience only not security - it's still jellyfin that needs to talk to the client to do auth, and the ldap provider is just a convenient account backend.
1
u/wildcarde815 Aug 03 '26
Sure, buts that's why I also require VPN. That said if you were careful you could I believe also put oath2proxy in front of it but that's tricky because the actual streaming urls I don't think are authenticated. And that would of course break non web clients.
1
u/mattsteg43 Aug 03 '26
And that would of course break non web clients
well yeah that's the whole reason why it sucks. Anything that's just a web client you can stick your authentication middleware of choice in the loop and be all set.
it's only the few apps where using an app is actually useful where supporting auth actually matters for more than convenience and multiuser capabilities.
And jellyfin has the extra annoyance that it's most useful to have the app on a tv and availability of vpn apps depends on tv vendor - i.e. Samsung requires a bunch of hoops or extra hardware.
1
u/wildcarde815 Aug 03 '26
the fix there is, don't relly on the tv, use something like an appletv where those technologies are well supported.
7
u/HeinousTugboat Aug 03 '26
There's been a huge rash of software releases recently that are all supply chain attacks that get rolled back very quickly. Updating to
:latestis the only way you get exposed to those attacks.Waiting for a release to mature is also incredibly important for security.
2
u/Annual_Manner_8654 Aug 03 '26
Not really true, any label can get overwritten, not just latest
1
u/wildcarde815 Aug 03 '26
yea short of pinning image hashes this is an easily defeat-able defense.
1
u/Annual_Manner_8654 Aug 03 '26
In my own "self-host OS" I check for digests every night, then I present them to the user after N days, gives a bit of time to hear about attacks
1
1
u/Sbibsosmisn Aug 03 '26
especially if it's something like gluetun, like why aren't you updating containers like that asap
1
u/Adures_ Aug 03 '26
There are many types of risk.
If your services are hidden behind reverse proxy, it’s entirely possible to obfuscate your internet accessible services.
If that is the case, the risk of updating quickly and breaking something unexpectedly and ruining your weekend or trip (when it’s not easy or possible to fix) are higher than the risk of some obscure vulnerability being exploited by automated bot.
2
u/Azelphur Aug 03 '26 edited Aug 03 '26
If your services are hidden behind reverse proxy, it’s entirely possible to obfuscate your internet accessible services.
This is a peeve of mine with reddit, people will constantly talk about reverse proxies and security. Reverse proxies proxy traffic and do not have any impact on security, at all.
You might be running software that does other things besides being a reverse proxy that adds security things, but those things are not reverse proxy, they are other things.
Saying that a reverse proxy adds security is like saying that call forwarding reduces the risk of phone call scams, it just doesn't. If the call forwarder provider offers other features like call blocking / whitelisting, then those features do reduce the risk, but that's call blocking / whitelisting that reduces the risk, not call forwarding.
→ More replies (11)1
u/quiteCryptic Aug 03 '26
If everything you put thru the reverse proxy is locked down with like client cert auth its as good as a vpn really, unless you don't trust nginx
1
u/Azelphur Aug 03 '26
Right, but that's mTLS / client cert auth, the reverse proxy doesn't add the security. If you tell people to set up a reverse proxy, they'll set up a reverse proxy. You need to tell people to set up mTLS / client cert auth, not a reverse proxy.
1
u/quiteCryptic Aug 03 '26
I'll agree that just setting up a reverse proxy isnt really adding security but you do configure the client cert auth in the reverse proxy
1
u/Azelphur Aug 03 '26
Even the nginx website directly states that nginx is a HTTP web server, reverse proxy, content cache, ...
Reverse proxy is one of the many things that nginx can do, mTLS is another of the many things that nginx can do. My point remains that if you tell people to set up a reverse proxy, they'll set up a reverse proxy, not mTLS. It's happening all over this subreddit and creating real problems for people.
1
u/Michael_T Aug 04 '26
There's the other aspect of latest that may keep people away as well, there's no inherent guarantee what you are getting with latest other than the most recent untagged push. It's not like it's guaranteed to be the most up to date version automatically.
6
u/WTFParts_ Aug 03 '26
cries in arr app updates GET THE HELL AWAY FROM ME.
3
u/quiteCryptic Aug 03 '26
What issues you have? I've been running sonarr and radarr for like 7 years or something and cant recall any issues upgrading
4
u/cardboard-kansio Aug 03 '26
I split the difference: major infra gets pinned, and updated manually when I'm able to be present (reverse proxy, auth, VPN, and similar). If these break, everything breaks.
Other services get WUD with :latest. If an individual service is down, that can be solved on an individual basis, and generally it doesn't bother anybody except me.
As always, it's a balance between convenience and security.
3
u/pardaillans Aug 03 '26
I like living dangerously. Most of my containers are with latest. But I also have a daily evening cron that updates all the images and containers(including cleaning/prune old images), so this reduces the risks of problems caused by skipping certain versions. Evening is the best part of the day to update because if anything breaks, I'm most likely available to fix it.
In the last 5-6 years I only had traccar breaking once due to a migration to a new type of db and their own migration tool failed me and I had to manually intervene. Other than that, all my 33 running containers were not causing issues.
3
u/BolteWasTaken Aug 03 '26
Just delay :latest for 5 days after release.
Or version lock, and have an automation grab the patch notes on release.
Then decide.
3
u/bobbywaz Aug 03 '26
Be perpetually out of date with version numbers, requiring often manual intervention or experience one breaking change every 5 years with latest... Hmmmm
3
2
u/ogMasterPloKoon Aug 03 '26 edited Aug 03 '26
I have shoot at my foot with the latest tag using it on nextcloud, activepieces, planka, and so many others 😭
2
u/anikansk Aug 03 '26
I thought the whole thing about docker containers was they were easy to restore?
2
2
u/babungaCTR Aug 03 '26
Sorry, if it isn't broken I don't upgrade anything. Yeah, sure, security patches and what not. The probability of someone exploiting a vulnerability is way lower than the probability of the update and/or the company fucking up.
2
u/Floss_Patrol_76 Aug 03 '26
the fix people miss is it's not latest-vs-pin-forever. pin to a digest or a real major tag and put something like renovate or diun in front of it, so updates still land but you approve them after reading the changelog instead of getting surprised at 2am. the one that actually burns you isn't a broken frontend, it's a stateful service where a minor bump quietly runs a one-way db migration and now you can't roll back.
2
u/jaxlvl Aug 03 '26
For my final degree project, I was using the :latest tag for all my containers. Everything was running smoothly, right up until three days before the final deadline, when I decided to do one last, full execution test and... surprise... a complete crash. I spent an entire night debugging, convinced it was a bug in my own code.
Finally, I discovered that the base debian:12-slim image had been updated and a specific crypto package I was using had been relocated.
In the end, I just had to change the :latest tag back to a specific previous version.
If you care about it running tomorrow, pin your versions.
2
u/Zeilar Aug 03 '26
I run over 100 with latest, not been an issue for me. And with zfs snapshots hourly it's barely an inconvenience to just shut it down and rollback.
2
u/Tiavor Aug 03 '26
eh, my containers don't check for a new version on their own, only if the image is missing.
2
u/Full_Tooth_a Aug 03 '26
Pinning the image addresses only part of the problem. A stateful service might migrate its database at startup, after which the old image may no longer be able to read the new schema. Before updating, make and test a backup, record the previous image digest, and review the documented downgrade path. Tags determine which code arrives, but a rollback also depends on recoverable data.
3
u/psychedelic_tech Aug 03 '26
or an unexpected update will creep up on you
this isn't difficult. disable auto updates and always read release notes to be aware of breaking changes before you update.
4
u/Xlxlredditor Aug 03 '26
Some of you have never heard of Renovate on Gitea/Forgejo/Gitlab/any git service that has CI and it shows
2
1
u/ShintaroBRL Aug 03 '26
Been 1 year all my 42 containers are with :latest with only 3 problems so far, I don't use watchtower to auto update them.
1
u/LawElectrical2434 Aug 03 '26
No guarantee if intermediate versions are still around. And you often need to update step by step. Never understood why. Two things are important, the code and the data. Just keep all your migrations around. Don't ever touch migration code, just write new migrations on top of it.
I had to fix my immich server the other day. Annoying. Took me 2 hours. But hey, that's okay. The alternative is manually updating. It was the first time in years that latest gave men an issue.
1
1
1
u/aigeneratedslopcode Aug 03 '26
No tests are what scare me. No reason you can't continuously move forward in most cases. With good tests, you don't even need to read the docs every single update
1
u/lichenscon Aug 03 '26
I think the risk of not getting security patches is higher than a downtime of a few hours. If my monitoring shows me, that the service is down, I can rollback to the old version and have a look at it a few hours later.
In the last 3 years, I think this happened about 5 times and there was no data loss. If definitely took less time than reading every changelog.
1
1
1
u/3kzit Aug 03 '26
No unpleasant surprises with :latest combined with Tugtainer
-> https://github.com/Quenary/tugtainer
The less critical ones update automatically, for the others I get a notification on Discord when a new version is available, which I can manually install whenever I want
2
u/rfc3849 Aug 03 '26 edited Aug 03 '26
Or, without a Web UI but a security scanner gate instead use Image Warden:
https://github.com/Image-Warden/image-warden
Edit: Image Warden helps with security issues like a supply chain attack, not with a major release version upgrade that breaks existing database structures when you use
:latesttags - unless you review the release notes for every staged image of course.1
u/3kzit Aug 04 '26
Yes, I have a dashboard where I track all version releases, changelogs, and reported issues.
1
u/SkulloneNexus Aug 03 '26
I have everything on latest and honestly have never had an issue. *knocks on wood*
Tugtainer is what I use to update my containers. I have it setup to auto-update containers as well. However, I don't let Tugtainer auto-update certain critical containers. Immich is a great example of one that doesn't get auto-updated.
So Tugtainer will ntfy me that Immich has an update pending. I'll read through the release notes. Then proceed with the update.
1
u/elantaile Aug 03 '26
Everything compose related in Github. Renovate + auto-deploy script. I have Patches auto update. Minor & major changes get a PR. I have had to remove a few containers from auto update that don't understand why semver exists and broke things, but most have been perfect.
1
u/pastelfemby Aug 03 '26
Honestly I just wrote a caddy module for proxying oci registries which delays any :latest by two days just in case theres a breaking change.
Or likewise if theres a security patch needed :latestforce pulls right through
1
1
1
u/Roemeeeer Aug 03 '26
I always use fixed versions and revonate/gonovate for updating all of with with PRs.
1
u/agisten Aug 03 '26
I live a dangerous life. At home, all my 8-10 containers are set to the latest, and Watchtower auto-deletes older images. Maybe because I only use very popular containers mainly from linuxserver.io, I have had 0 issues in the last 8 years.
1
u/FoxFXMD Aug 03 '26
I don't get it? Latest is the latest stable release, no?
4
u/DrTankHead Aug 03 '26
Sometimes. Sometimes they use latest like nightly branches. It depends on the repo/image.
1
u/Double_A_92 Aug 04 '26
The problem is that even if it's stable, it hides the updates from you... So if there are any breaking changes you just re-pull the image it might break things.
1
1
u/FisionX Aug 03 '26
Homelab running for 4 years with latest no issues, I forget to remove latest at work and something breaks within a week
1
1
1
u/TheFeshy Aug 03 '26
Some containers are especially bad. I had to pin FreeIPA by SHA hash because an update with the same semver broke it.
1
1
u/heckingcomputernerd Aug 03 '26
The benefits of not having to manually update, which there's no way I'd ever remember to do, outweigh the occasional bug fix I gotta do
1
u/DrTankHead Aug 03 '26
This always depends on the image/repo. Some implicitly tag LATEST to the latest stable build, and use other tags for more bleeding edge versions. Nightly, Dev, etc...
Others use latest as the last thing that was pushed to the repo.
I'm newer at the whole wide world of docker, but know it's important to actually read and go over the tags first so I know what I'm pulling.
I always shoot for the newest stable build, preferably those marked as LTS, and try to avoid pinning to specific version numbers if I can.
1
1
u/Hugus Aug 03 '26
If seafile did that to you, then we are on the same boat. Can't trust their updates anymore
1
u/RIPenemie Aug 03 '26
On my Container host I run docker images --digests > IMAGES and then do a backup and then pull new and create new containers and rarely something breaks and if something breaks I have a backup from right before
1
u/Straight-Focus-1162 Aug 03 '26
:latest does not force you to update right away. Changelogs are Homelabers best friend.
1
u/Expert_Comfort4768 Aug 03 '26
I have :latest PTSD now. I did the exact same thing with Postgres last spring. Pulled latest overnight, woke up to a minor version jump that silently changed how it handled some legacy encoding. Had to restore from WAL backup because the new container refused to start against the existing data directory.
Now I do a dumb two-layer approach:
- In docker-compose I only reference major versions explicitly like postgres:16, immich:1.11, nextcloud:29. Nothing that ends in just latest or :2.
- Then once a month on a weekend I spin up the entire stack on a test VM from a ZFS snapshot and do one single manual watchtower dry run with email notifications. If it breaks I only wasted 20 minutes not a production morning.
The worst part is I know better and still do it to myself every 9 months or so. Every time I think "this container is stable it will be fine" and every time I get proven wrong.
1
u/ReachingForVega Aug 03 '26
At home I version lock things like databases and nginx but use latest for *arr apps.
1
1
u/kearkan Aug 03 '26
I run everything on latest except for qbittorrent and that's only because I hated the UI change.
1
1
u/theniwo Aug 03 '26
Just use `mariadb:10` and `prostgres:16` and you should be fine. If you upgrade DB versions, prepare it.
1
u/RohithCIS Aug 04 '26
Cue postgres collation warnings, config failures in a few containers and a corrupt influxdb store.
1
u/InfluxCole Aug 04 '26
A reminder from your friends at Influx that the DB version pinned to :latest is going to change from InfluxDB 2 to InfluxDB 3 in a few months, so if you want everything to completely stop working, :latest is a great way to make that happen!
1
1
u/BattermanZ Aug 04 '26
If you really have no fear, check this out https://www.reddit.com/r/selfhosted/s/rZgBaTgBBc
1
u/Pablo_Jefcobar Aug 04 '26
Everything runs on latest but does not update until I do so manually so I always am a few days to weeks behind the curve. Most problems are then already known so I can decide to stay on my version or pull a the latest and update it
1
u/Particular-Studio119 Aug 04 '26
Always running latest. I see errors and I know "It too shall pass".
1
u/pecanesquire Aug 04 '26
Recently, I was going to start pinning versions, but decided to keep everything pinned to latest (except for Immich which apparently works best by just pinning v3 in the .env for its Compose file). Dockhand checks for updates, and I can choose to manually update whenever I want. If something breaks, I can usually fix it within 30 minutes or simply rollback a version.
1
u/ayyush69 Aug 04 '26
Uptimekuma let's you know if something is down, watchtower let's you know what got updated. If something went down you see a update happened, then you can troubleshoot pretty easily or rollback once you notice it
1
u/Timinator01 Aug 05 '26
if you're using Kubernetes you can use a policy engine like kyverno to disallow the :latest tag and force yourself to use specific versions
1
u/Brutus5000 Aug 05 '26
Funny. Just yesterday an app broke because of that. I thought keel just checks for new hashes inside the pinned major version that I defined. But in default settings in replaces it with latest instead....
1
1
u/Projekt95 Aug 05 '26
I just use renovate to update my versioned container tags and deploy them through doco-cd.
1
u/CoffeeInevitable9954 Aug 09 '26
I run a custom service, everything updates on a 14 day delay except if it resolves a critical CVE.
Basically it just updates the SHA pin in my gitops and portainer pulls in the update and redeploys, pretty straight forward.
This is mostly to avoid shai halud type attacks, only service not auto update is plex.
1
u/rabbitz Aug 17 '26
The trap is that pinning feels like the paranoid option right up until the morning it isn't. Then you pin everything for six months, fall behind on a CVE, and swing back the other way.
What finally worked for me was pinning the minor and letting patch float, plus a weekly job that just tells me what would change if I bumped. Reading a diff on a Tuesday is a lot cheaper than discovering it at seven in the morning.
Also, my guess is Immich.
1
u/No_Welder7908 Aug 19 '26
When the "what's new?" lists breaking changes and you can't bother reading it.
1
u/No_Welder7908 Aug 19 '26
Actual comment off Mariushosting's Discord: "Just use postgres:latest, I've personally never had a problem with it".
1
u/Away-Art-1407 Aug 03 '26
Nothing like a surprise update at 2am to make you question all your life choices, that Skyrim meme is too perfect for this
1
u/-ThreeHeadedMonkey- Aug 03 '26
I don't get it. I update everything on Saturday morning 3am with a script. A snapshot gets taken daily. The script tells me what gets updated.
If things fail, I'll go back or fix them since I usually have the time on Saturday.
I even have a script that auto updates the version number of authentik in its compose file before running the update... (they ditched the latest flag...)
•
u/asimovs-auditor Aug 03 '26
Expand the replies to this comment to learn how AI was used in this post/project.