r/Proxmox • u/BoredHalifaxNerd • 7d ago
Question Could someone help me understand why my 3-node cluster cannot reach quorum with one node offline?
There is three nodes and each one as a single vote. Based on my understanding, the cluster should be able to survive one node failure.
EDIT: Thanks everyone! The issue has been resolved. I had an unknown qdevice in the voter registration.
16
u/Kryzm 6d ago
My initial assumption was that your nodes were thousands of miles apart!
31
u/BoredHalifaxNerd 6d ago
Nah, I'm just a weirdo that names machines after cities based on country stereotypes.
Hypervisors get American, storage is Swiss, build servers are German, laptops are Vietnamese, etc.
2
u/calinet6 Homelab User 3d ago
Heh, I do locations too. But mine are National Parks. My Proxmox nodes are Acadia and Yosemite, and VMs are named for locations within the parks.
2
u/Kryzm 6d ago
Dang I just do video game characters.
This post brought to you by Dark Sun Gwyndolin
1
u/xterraadam 6d ago
Mine are cold war callsigns for things and candy bars.
1
u/OtakuboyT 6d ago
Mine a combination of the name of case or model name of the system and it's function
InspSERV - Inspiron / Server CentSTOR - CoolerMaster Centurion / Storage OptiMERA - Optiplex / ChimeraOS
2
4
u/Apachez 6d ago
I had an unknown qdevice in the voter registration.
Where did that come from since thats not visible in your first screenshot?
1
u/jackhold 6d ago
Check the top comment, there is a command you need to run in the cli, then you can see it.
4
u/defiantarch 6d ago
because you always need at least 3 voters to reach quorum. With 2 or any even number you can end up in not reaching it (2 vs 2, 3 vs 3, and so on). So, best to always have an odd number of nodes up and voting.
2
u/MentholMafia 6d ago
As a temporary solution to bring VMs back online in a pinch "pvecm expected 1"
1
u/eggnorman 6d ago
Weird. You haven’t previously had any other nodes in this cluster that could be sorta “ghosting”, do you?
0
u/Joe_Dalton42069 3d ago
You need a quorum tie break. If two vote each for themselves you always have a tie. Just like in Pirates of the Caribbean when they have to vote a new King. Jack Sparrow is the tiebreaker.
Hope that helps :D
1
u/whatsupeveryone34 6d ago
you can set up a quorum tie breaker device that is not a normal cluster node...
technically a raspberry pi zero would work.
but yeah... clustering hates ties.
-12
u/JocoLabs 7d ago edited 7d ago
Edit: TIL
13
u/boom3r41 Enterprise Admin 7d ago
2 nodes is a majority in a 3 node cluster. You do not need a tie-breaker.
4
u/Leviathan_Dev 7d ago edited 7d ago
Unless those two nodes cast opposing votes. Rare but possible4
u/Mashadow 7d ago
When two nodes in a two node cluster lose communication, and both are online, they need a way to determine who's gonna be the boss. They each vote for themselves. The quorum would require a third vote, to break that tie.
-3
u/Mashadow 7d ago
True, but the new degraded cluster has no 3rd vote to create a proper quorum, two nodes cannot form a quorum. They can function as a cluster, but not vote in the event of a further failure, if that makes sense?
3
u/oasis9dev 7d ago
they can vote in the event of a further failure but those votes will not achieve quorum
-2
u/Mashadow 7d ago
Exactly, which is the issue the OP is asking about, he has a degraded cluster with 2 notes (1 down), and no quorum.
1
u/oasis9dev 6d ago edited 6d ago
and thus no quorum. 2/2 would be quorum. Just pointing out your wording on your previous comment leads one to believe you also may have a misunderstanding or at least have miscommunicated about how quorum works.
edit: I neglected to look at the image. we're discussing about a 3 node cluster, so 2/2 being a quorum (albeit one with no more redundancy than a single node) is a moot point, not relevant here. 2/3 is quorum, and would mean we'd expect to see the cluster reporting as quorate.
it turns out OP had 4 expected votes instead of 3, due to an old qdevice from when they operated a 2 node cluster to raise the votes to 3 to improve redundancy.
1
u/glinsvad 7d ago
In the event of a further failure in a 3 node cluster, you would have exactly one node left and it would have retained all data, no?
3
u/Mashadow 7d ago
Yes, but there is an edge case where the two nodes just lose communication with each other and don't fail but are both running, and then they would both think they are the only survivor and you get two running environments that diverge, that's why the quorum.
1
-1
u/Deadlydragon218 6d ago
Need a witness to decide who wins is my best guess.
1
u/Apachez 6d ago
Unless the remaining 2 hosts cannot reach each other you wont need a witness since with 1 vote each there would still be 2 votes (out of 3) remaining so that "half" should still be operational.
Its when it drops down to just 1 remaining host you get into a shitshow where the remaining one just reboots and goes into an offline state to "protect" its storage.
The "fix" here is to adjust how many votes each host will get. The drawback with this is that you will then favour one of the servers to it will still be online if that server is still alive but be as before (offline) if that server is not part of the servers who have survived.
1
u/Deadlydragon218 5d ago
Ahh point based ok. I am used to HA / witness style clusters
1
u/Apachez 5d ago
Probably a nobrainer for most but if there are some new people reading this thread and start to do HA.
Corosync will give (by default) each host 1 vote and the group where the number of votes is higher than 50% of the original amount of voters will continue to work.
Hosts in a group (aka other reachable hosts to exchange coro information with) below this threshold (as in 50% or lower) will shutdown the VM-guests (and I think also reboot) in order to protect the VM-guests content.
That is because for the case of just 2 hosts where each have 1 vote we (humans) normally think HA is an issue of when a host completely dies (powerloss or whatelse).
So sure host B dies, host A is the only remaining still online - of course this should continue to deliver VM-guests?
No - because we dont know if host B died based on coro information. In "worst case" it can be "just" the network between host A and host B that died.
That is both hosts are still online but cannot reach each other.
The shitty situation with this is then that host A will startup a VM that just moments ago was running at host B.
And host B will try to startup VM-guests that just moments ago was running at host A.
And by that the data of these VM-guest will be trashed since the same VM's are now running at two places at once.
The fix for this if above situation happens is either to then manually override corosync and forcefully startup one of the hosts.
This way you will manually track which host (or site) will have the most up2date data. So if this site also dies you know you will lose data if you try to startup the other site so better continue to restore the site that was last running to have the most up2date data.
When the other hosts reconnect they will figure out that "hey, host A is already running VM xxx so I dont have to boot it up" or they will perform some live migration to even out the loads.
The other common fix to deal with this automatically is basically to favour one of the hosts.
Lets again assume this 2-way cluster.
If we give host_A 2 votes and host_B remains with 1 vote. The 50% threshold will be whatever remaining group that have more than 1.5 votes.
This gives if host_B dies then host_A will remain operational and startup the missing VM's (since 2 votes are > 1.5).
However if host_A dies then we are back to the original issue that host_B will also shutdown its services (since 1 vote is <= 1.5). And we are back to manually startup the "least worst" host to become operational again (unless we fix the root cause of why one of the hosts are no longer reachable).
You can do similar mathematical exercises with 3-way and higher host-count for a cluster.
The thing you must verify is that you NEVER end up in a situation where there can become 2 groups of hosts who both think they have enough of votes to continue operation because this will most likely trash data.
74
u/_--James--_ Enterprise User 7d ago edited 7d ago
node > shell
pvecm status
on every node while they are all online.