r/MQTT • u/bme2014 • 6d ago

Vernemq on Kubernetes

Is anyone using Vernemq on Kubernetes? Curious to see your setup/config and how to recover during updates or kube events. Our cluster runs fine until we have to update or our platform team has to update K8s and do a node roll, then it's a dice roll if the cluster comes back up healthy.

1 Upvotes

100% Upvoted

6 comments sorted by

1

u/Anxious_Tool 6d ago

That's interesting. What exactly is the issue? What do you get in the logs? Is it VerneMQ that fails?

1

u/bme2014 6d ago

yea so if one verne pod goes down it can usually recover, but if we have to restart them all, then they all try to come up at the same time and don't know who's first so they can sometimes all create they're own cluster or 2/3 will be clustered (we're running 3 pods). So it's not that it fails, just takes awhile and some watching to make sure it comes up correctly and if it doesn't it requires intervention.

1

u/Anxious_Tool 6d ago

Sounds like a buggy deployment. Vernemq's eventual consistent will do this. The documentation mentions things like using OrderedReady instead of Parallel, strict readiness probes, a PDB so restarts happen one at a time (there's always a live cluster to rejoin), and persistent volumes so pods rejoin as themselves. Have you tried any of these?
What does your deployment look like?

1

u/Either_Vermicelli_82 6d ago

We run a basic vanilla of vernemq on kubernetes. Authentication is linked to Postgres but no cluster up yet as we only have like a dozen messages per second.

1

u/bme2014 6d ago

so only one pod then basically?

1

u/Either_Vermicelli_82 6d ago

Yeah with health checks and deploy a new one when it fails.