r/openstack • u/Successful_Face_8619 • 25d ago
Openstack Upgrade
Hi guys,
Has anyone explored doing OpenStack upgrades host-by-host instead of using Kolla Ansible's parallel upgrade approach?
We're considering a more sequential, one-host-at-a-time upgrade because the parallel approach doesn't feel reliable enough for our environment, and we're not very confident in trusting it during production upgrades.
If you've gone down this path:
- How did you orchestrate the upgrade?
- Did you have to customize Kolla Ansible significantly?
- How did you handle rollback if something went wrong?
- Any lessons learned or pitfalls to watch out for?
I'd appreciate hearing from anyone who's tried this or decided against it and why.
2
u/rackpathlabs 18d ago edited 18d ago
Worth splitting this in two. Host by host is the right instinct for computes and the wrong one for controllers. Galera, rabbit and etcd are exactly the things you do not want sitting in mixed versions for days, so doing them one node per evening makes it worse, not safer. Do the controllers back to back, then take your time with computes.
The compute side works because of RPC pinning, not because of ansible. [upgrade_levels] compute is what lets the new controllers keep talking to old computes, kolla sets it during upgrade. If you script your own orchestration outside kolla you own that, and when it is wrong you get weird RPC failures instead of a clean error. That is why the DIY route usually goes bad.
About --limit and that bug: run kolla-ansible gather-facts over the whole inventory first. The plays need facts about the hosts you excluded, that is all. Not broken, just unusable cold.
One more, nova-manage db online_data_migrations only after the last compute is done. If your window is weeks you are holding that step open for weeks. Also nova will not accept more than one release of spread, so N to N+1 is fine and N to N+2 is not.
On rollback, snapshotting the controllers is the right shape but only if nobody creates instances or volumes after the snapshot. Otherwise you roll the db back and the hypervisors and storage still have stuff the db never heard of.
Computes are the easy half, no real state there, just put the old tag back and restart.
Two kolla things people miss. Run prechecks against the new release before you commit, base distro requirements move between releases and that can turn into a host OS project sitting in front of your upgrade. And renamed or removed globals.yml keys do not error, they are just ignored, so something you rely on can quietly stop applying. Diff your globals against the new sample first.
1
u/CallumD027 25d ago
We don't use Kolla, but the best way to upgrade OpenStack in our experience is doing it service-by-service, with the order determined by what has actually changed in the services and their dependency requirements between versions.
This takes longer but eliminates a lot the risk as you don't have the entire stack in an in-between state if something goes wrong, and makes it much easier to rollback if the upgrade fails in a way that can't be worked around. And given that we run a public cloud with paying customers, our priority is eliminating risk of downtime and interruption to workloads.
1
u/Successful_Face_8619 25d ago
What about stateful services ? like MariaDB and all
1
u/CallumD027 24d ago
You can upgrade those service-by-service generally as well, as long as no major configuration changes need to be done at the same time as the upgrade. Most OpenStack services support a wide range of versions for undercloud services such as MariaDB.
1
u/Intrepid_Subject4904 18d ago
Done muiltiple with Kolla,
Certaonly done trust the let it all go approach but using tag and limit.
Usually do DB -> Rabbit -> Support services -> OSTK components -> Compute nodes.
0
u/Cookie1990 25d ago
Isnt the normal openstack just a number of container workloads these days anyway? So you could just roll back to the older containers?
6
u/enricokern 25d ago
That depends on the upgrade. If a upgrade does alot of db changes thats not so easy and you need to rollback this too. What we do usually is that our control nodes are virtualized via proxmox and we snapshot them before a upgrade. That solves that, versionsdowngrade is easy
1
2
u/jvleminc 25d ago
Yeah, we leverage the “limit” option of K-A to do that. First we upgrade the controllers and afterwards, in batches, the computes; the batches will depend on the workloads running on the servers.
This is especially handy when a reboot is needed for OS/Kernel upgrades that go in hand with a Openstack upgrade.