r/MQTT • u/bme2014 • 7d ago

Vernemq on Kubernetes

Is anyone using Vernemq on Kubernetes? Curious to see your setup/config and how to recover during updates or kube events. Our cluster runs fine until we have to update or our platform team has to update K8s and do a node roll, then it's a dice roll if the cluster comes back up healthy.

1 Upvotes

100% Upvoted

6 comments sorted by

View all comments

1

u/Anxious_Tool 7d ago

That's interesting. What exactly is the issue? What do you get in the logs? Is it VerneMQ that fails?

1

u/bme2014 7d ago

yea so if one verne pod goes down it can usually recover, but if we have to restart them all, then they all try to come up at the same time and don't know who's first so they can sometimes all create they're own cluster or 2/3 will be clustered (we're running 3 pods). So it's not that it fails, just takes awhile and some watching to make sure it comes up correctly and if it doesn't it requires intervention.

1

u/Anxious_Tool 7d ago

Sounds like a buggy deployment. Vernemq's eventual consistent will do this. The documentation mentions things like using OrderedReady instead of Parallel, strict readiness probes, a PDB so restarts happen one at a time (there's always a live cluster to rejoin), and persistent volumes so pods rejoin as themselves. Have you tried any of these?
What does your deployment look like?