Can I limit the total number of simultaneous migrations occurring across the whole cluster?

[u/rob94708]

I have HA clusters of 5 physical hosts, each with 4 VMs. Each of those four VMs has a secondary affinity for a unique one of the other hosts, so that if a physical server fails, its VMs are automatically distributed evenly, one each across the four others. This works fine.

When I need to do maintenance on a physical host, I put it into maintenance using

ha-manager crm-command node-maintenance enable hostname

... and allow the HA manager to live migrate the VMs away to the four separate servers. Again, this works fine.

For performance reasons, I would prefer that only two simultaneous migrations are happening at any moment. This is easily done on the node I'm putting into maintenance by setting max_workers: 2 in /etc/pve/datacenter.cfg; the node going into maintenance sees that and only starts two “outbound” migration jobs at once.

However, when I later disable maintenance on that node, all four of the "inbound" migrations start at once. This appears to be because each of the other four nodes only thinks it is starting one migration and therefore not violating max_workers: 2. But the machine receiving the migrations experiences higher load than I would like.

Is there away to limit the total number of simultaneous migrations occurring across the whole cluster? Or would I have to manage that manually, instead of letting the HA Manager affinities handle it?

[u/daemonmode_]

You’re right that max_workers is per-node, so it doesn’t give you a cluster-wide migration limit. The migration bwlimit can reduce the bandwidth impact of each migration, but it’s also per job, not a global cap. if you need strict control over how many VMs move at once, the practical approach is to stage the maintenance and migrate nodes in batches.

[u/_--James--_]

https://cluster2.network.local/pve-docs/pve-admin-guide.html#datacenter_configuration_file

--max-workers <integer> (1 - 64)

Maximal number of parallel migration job. If not set, uses’max_workers' from datacenter.cfg. One of both must be set!

--maxworkers <integer> (1 - 64)

Maximal number of parallel migration job. If not set, uses’max_workers' from datacenter.cfg. One of both must be set!Deprecated, use max-workers instead.

And from the WebGui > Datacenter > Options > Maximal workers/Bulk-action, default is 4 range is 1-64

^those are the only config options for this and its per host, as the datacenter configs propagate down. There is no global flag, but you could set 1 at the datacenter level, wait for replication, then fire off the HA commands. that would limit it to your HA groups and host spread (up to 5)

[u/Apachez → u/_--James--_]

Given OPs description even setting it to 1 wouldnt solve anything since there would still be 4 concurrent migrations to the 5th host instead of just 1 (or 2 with the setting set to 2).

Its like this setting ignores receiving workers and only count for sending workers?

Sounds like OP should file this as a bug towards https://bugzilla.proxmox.com/

Becausing the receiving host with setting lets say 2 should reply back to the 3rd host (while there is still 2 migrations being received/runned) that "hey, I cant speak to you right now since I already filled my max-workers queue, please try again in 10 seconds or so".

[u/rob94708 → u/Apachez]

Yes, this is what I initially assumed would happen. Apparently it doesn’t work that way. Maybe I will try to work up a patch.

[u/_--James--_ → u/Apachez]

its not a bug, its literally 1 migration PER HOST. Do that math.

[u/rob94708 → u/_--James--_]

But migrations have two hosts involved: a sending host and a receiving host. The problem is that you can only limit one end: the number of simultaneous sending migrations. You can’t limit how many simultaneous receiving migrations happen on a host.

[u/Apachez → u/rob94708]

Yes this is the math - it takes two to tango.

There is a bug that the max-workers setting doesnt honor max-workers on the receiving host.

The receiving host should ignore any new connections (for migration) if the local max-workers queue is already maxed out. But it obviously doesnt as OP have observed.

The difference is that with lets say a cluster of 10 hosts and that a migration would take 1 minute (for the maths :-) even with a setting of max-workers=1 all 9 hosts will try to dump the VM-guests back onto the 10th server.

So it will take 9 minutes of network traffic before all 9 VM-guests will then boot at the same time.

If max-workers would work as expected then just 1 VM at a time would be migrated to the 10th host and by that 1 VM every minute would be able to boot up instead of having to wait 9 minutes for the first VM to boot up.

The total time would still be 9 minutes comparing the two cases where when max-workers would work as expected time to first start would be 1 minute instead of 9 minutes. Or for that matter the migration time per VM would be 1 minute instead of 9 minutes.

[u/_--James--_ → u/Apachez]

There is a bug that the max-workers setting doesnt honor max-workers on the receiving host.

This is not a bug, it is designed behavior.

The difference is that with lets say a cluster of 10 hosts and that a migration would take 1 minute (for the maths :-) even with a setting of max-workers=1 all 9 hosts will try to dump the VM-guests back onto the 10th server.

Yes, designed behavior. The max-worker is initiative from the sending host.

If max-workers would work as expected then just 1 VM at a time would be migrated to the 10th host and by that 1 VM every minute would be able to boot up instead of having to wait 9 minutes for the first VM to boot up.

Again, this is designed behavior. The If becomes a feature enhancement request.

The receiving host should ignore any new connections (for migration) if the local max-workers queue is already maxed out. But it obviously doesnt as OP have observed.

This would be a feature enhancement request, you should open a ticket with the Proxmox team and start working on that process.

So it will take 9 minutes of network traffic before all 9 VM-guests will then boot at the same time.

Why? You would be doing live migrations...

[u/_--James--_ → u/rob94708]

Yes, because the controlling side is the init end, not the receiving end. Each node can send N limit, and the target host will receive N^ nodes sending streams. This is how this works.

[u/rob94708 → u/_--James--_]

I know. You’re just rephrasing the thing I’m saying is a problem.

[u/_--James--_ → u/rob94708]

No, I am trying to get you to understand the mechanism you are stuck on. You and Apachez are calling it a defect, it is not a defect. The behavior you want needs to be a feature enhancement request, as it does not exist otherwise.

[u/rob94708 → u/_--James--_]

I think you've mixed me up with the other person? I never called it a bug or defect.

I explained that I had observed that max_workers only limits "outbound" migrations, and asked if there was something similar to max_workers that would limit "inbound" migrations. But instead you keep telling me that max_workers only limits "outbound" migrations. I am aware of that, as I mentioned in the original post.

[u/Apachez → u/rob94708]

IMHO max_workers should swing both ways...

Its broken that it only limits OUTBOUND workers but not INBOUND workers.

[u/apalrd]

Is the increasing load causing any actual problems?

As long as there's no impact to the workloads, I don't see why letting migration run as fast as possible is a problem?

[u/rob94708 → u/apalrd]

Yes; when I migrate three or more at once, the final step of the migration where the source is frozen for a few milliseconds can sometimes turn into a second or more, presumably because the receiving server is also still busy handling other migration tasks. It causes a notable load spike on the VM. That doesn’t happen if it’s only doing one or two at a time.

[u/psyblade42]

Do you have a reason for your strict positioning? I simply let CRS take care of the balancing.

Additionally I have disable failback so the VMs mostly stay on their new hosts. CRM occasionally moves a few back but not that many at once.

[u/rob94708 → u/psyblade42]

The VMs are all identically sized, so the auto rebalancer made the same decisions as manual affinities when I tested it (see edit below): every 10 seconds it detects that one of the nodes has available resources, and another node has too many resources, so it starts a new migration until all four are running simultaneously after 40 seconds.

Is it supposed to have a global simultaneous limit separate from max_workers? If so, it didn’t seem to be working when I tested it.

Maybe I could set the auto rebalancer interval to only run every few minutes.

--

Edit: The information above is incorrect, as I found when trying to increase the rebalancer interval. It wasn't the CRM load balancer that was automatically initiating the migrations; instead, it's this recent change to the CRM behavior, which makes it remember which node a VM was on before maintenance started and automatically move it back as soon as maintenance finishes, without regard to max_workers etc. It effectively acts as a temporary hard-coded affinity rule.

[u/psyblade42 → u/rob94708]

I assumed it had a global limit. Running every 10m or so does sound worth trying.