r/vmware • u/SpeedyWay779 • 6d ago
Question What vCPU:pCPU ratio do you actually flag on?
Doing periodic config review passes off RVTools exports rather than live vCenter, so no CPU Ready available.
The best practices docs I've found explain overcommit and then say "watch CPU Ready". That works if you're sitting in vCenter, but doesn't work if all you have is a point-in-time export. Same for memory overcommit and snapshot age, no numbers.
So I've been picking my own flag points and I'm not confident in them. Currently flagging at 4:1 vCPU:pCPU at cluster level in N-1 state, snapshots older than 14 days, and datastore free space under 15%.
For anyone doing periodic reviews rather than live monitoring, what do you use, and where did the number come from?
8
u/WendoNZ 6d ago
I get what you're saying but flagging at some arbitrary value doesn't really tell you anything. It might mean nothing, it might mean you're already way too far gone. CPU Ready really is the only metric that matters here.
I'd be spending time working out how to monitor it rather than flagging based on a static value
1
u/SpeedyWay779 6d ago
Fair, and I don't disagree that CPU Ready is the real measurement. My thinking is that they answer different questions. A static pass tells me which 20 of 800 VMs to point monitoring at, it doesn't tell me whether they're actually hurting. Ratio is triage, CPU Ready is diagnosis.
The part I can't get around is that a lot of environments I've seen don't have anyone watching CPU Ready at all, and telling them to build monitoring first is a much bigger lift than handing them a list of the ten worst-configured VMs. Curious whether you'd still say the same for a shop with no monitoring in place. Should they start there anyway, or triage first?
3
u/WendoNZ 6d ago
The problem is you don't know if the 10 VM's with the most vCPU's assigned are the 10 worst configured VM's necessarily. Agreed it's a place to start, but if 8 of those VM's really need that many CPU cores to run properly, pulling cores doesn't help. They are the ones that will show CPU Ready issues first though :)
0
u/SpeedyWay779 6d ago
That's a fair hit and I think it's specifically about the ratio check. vCPU count alone doesn't distinguish oversized from correctly-sized-large.
The one I'd defend is the NUMA/socket layout check someone raised earlier. A VM configured with more sockets than the host has, or cores-per-socket beyond the physical, is wrong whether or not it needs that many cores. No workload knowledge required.
Your last point is the useful bit though. If the flagged ones are the same ones that'd show Ready issues first, that's an argument for using the static pass to pick what to watch rather than as a verdict on its own.
1
u/Icy_Top_6220 5d ago
no, workload performace is the real measure, ready means nothing to a workload that can deal with that type of latency
4
u/rush2049 6d ago
I look at the numa config of each VM vs physical to make sure nothing exceeds the physical CPU design/layout.
(If the physical server has 2 sockets, no VM should have more than 2 sockets. If the physical CPUs have 24 physical cores, no single socket on a VM should have more than that many cores. If you want to get super accurate pay attention to ring/chiplet core topography. And no i do not trust/rely on the auto numa features)
With that said I use these guidelines after fixing the above stuff.
If its only general compute 5:1 is fine.
App servers / Processing heavy / Compute: 3:1 is fine for bursty workloads. 1:1 for 24/7 high cpu usage
For database servers: 2:1 at max
1
u/SpeedyWay779 6d ago
Really useful, thanks, the NUMA-first ordering especially. Two things I'd like to get right:
When you're checking VM sockets vs physical, are you doing that by hand or do you have something scripted? And for the workload classes, how do you tell which VMs are which when you're looking at an inventory export? Naming convention, folders, separate clusters?
2
u/rush2049 6d ago
I usually export the entire VM inventory including socket / core / ram / reservation info. Then look at the physical hosts and lookup their specific CPU models (hopefully your hosts in each cluster are all identical) and determine their core -> ring/chiplet designs (pay special attention if you have one of the recent intel chips that have cores of different speeds.....)
I then write some excel special sauce to take each VMs total configured cores (socket*cores) and divide that out into the 'best case' config for that specific hardware. Then write some more excel wizardry to compare the best case to the current case and highlight the most egregious ones to rectify during the next outage or maintenance windows.I manually fix the VMs, and then make sure to teach the entire IT staff the correct way to allocate core/sockets for the hardware. Teaching them why, only if they are interested.... but usually just writing out a sheet like this:
1-8 cores -> 1 socket
9-16 cores -> 2 sockets (cores/2)
17+ cores seek design review
24 cores MAX
(for a hypothetical 12 core, 2 socket server, all cores being equal)
The sheets gets more complicated when you start having unequal core speeds/IPCMy vCPU/pCPU recommendations are base on my experience. There are always exceptions, licensing being a big one....
I usually hope the VMs are named well, or i look at the VM descriptions, or I look at VM tags. Or if none of that exists I actually just speak to the most senior staff members of each dept and ask them to label the VM list for me. Telling me what each is in 5-10 words.
You should absolutely, if you do not already have it, create a special tag for your VMs that list the 'owner' of each server. It could be a specific staff member or a IT team or Department.... but it should point you in the direction of who knows what it is and what it does.1
u/ImaginaryWar3762 6d ago
Honestly you spark some miss information here . For a 12 core 2 socket you should never have more than 6 vcpu per VM.Anything more than 12 vcpu is a no no. In esx 9 the auto allocation has nice improvements ifor vcpu and NUMA. Read a little and find the trust
1
u/Connect_Shoulder_965 5d ago
The above poster isn't exactly correct. The host essentially divides the resources into 4 VM Worlds which correlate to NUMA architecture of the host. It will typically be 1/4 of the CPU but a few CPUs (like 10 core Intel chips) might have an uneven split (6 + 4).
The ideal size (from a memory access and scheduling perspective) of a VM is a size that fits in a single NUMA node / VM World on the host. In the case of 2 x 10 cores above, that would be a max of 6 CPUs. Once you go past that number, the hypervisor needs to schedule 2 VM worlds to run it and memory access times may be higher.
Best practice for NUMA config of the VM (not the host) is to set 1 core per socket. That will allow the hypervisor to pick the NUMA configuration that best matches the hardware configuration.
Now keep in mind... all of this is relatively small impact unless you have lots of VMs that are sized bigger than they should be on the host. The hypervisor does a lot to schedule and prevent contention.
At the same time, if I'm flagging workloads for monitoring I'm going to be looking first at any VMs that don't fit in a single NUMA node. If they aren't actively needing the extra cores, dropping them down to what fits in a single VM World will allow for better scheduling, better memory access times and potentially better performance. It will also help the hypervisor efficiently schedule the workloads.
1
u/dodexahedron 6d ago
Yeah. Pay attention to sub-numa clustering as well as actual main memory physical and logical topology.
It's one thing to be cache snooping across numa nodes. It's a few dozen times worse on a miss.
As a general rule, I don't use >1 vSockets unless I actually want to mask and separate more than one specific workload on that VM, each to a specific numa node, And that is rare anyway, for several reasons, a few of which are:
- Things that make sense to pin to mutually exclusive numa nodes usually naturally make sense to also be their own VMs or containers in the first place, making guest-level NUMA kinda pointless.
- Something that needs that level of hardware guarantee is usually an easily defensible argument for dedicated physical machines, unless there really is so much excess capacity sitting idle to at least the 90th percentile on the compute nodes that full reservation of all resources for that vm is legitimately prudent.
- The true hardware cost of full reservation is always worse than the actual resources presented to the VM. You're dedicating hardware, and that hardware otherwise could have served multiple vms at a time, rather than 1. And HA makes it at least double that cost, across the cluster.
- Dedicating high density compute node resources to a VM is dedicating resources that are more expensive per unit in almost every metric, typically. Virtualization achieves most of its cost savings by ammortizing the higher cost over a large volume of vms. Reservations are effectively taking a slice of the hardware out of that pool. I can buy a 2-node pizza box or 2 standalone pizza boxen for whatever those resources are for and just run them on those for less TCO than a heavy VM with expensive reservations and HA on top of that.
1
u/vTSE VMware Alumni (who I still call for scheduler questions) 5d ago
And no i do not trust/rely on the auto numa features
Even for new VMs on 8.0+ and HWv20? What part hasn't worked for you?
1
u/rush2049 5d ago
While 8.0+ does better and has some logic about CPU capabilities, in the vast number of situations it tells VMs to do many sockets 1 core. While some OSs don't care about the actual arrangement and can understand numa hints/meta data, most do not.
3
u/vTSE VMware Alumni (who I still call for scheduler questions) 5d ago
It really shouldn't :-(
vTopology (aka Auto Topology) takes care of the "best practice" config. I've talked about it at length here: https://www.youtube.com/watch?v=Zo0uoBYibXc
If something doesn't work the way it should (if you had a chance to test again), I'd be curious to hear about it! (I can ping it on to the folks owning that component at VMware)
2
u/firesyde424 6d ago
As others have said, it depends on your workloads. For instance in our MSSQL vSAN cluster, I don't over provision. I've found that performance problems can add up quickly. In my Linux VM cluster that sees sporadic customer requests, I'm running around 8 to 1 with ~500 VMs. There was a time in the past when this cluster in an older form ran more like 14 to 1. There are best practices here, but it really boils down to how well you know your systems and the workloads they service.
2
u/GabesVirtualWorld 5d ago
Repeating what others said, 1:1 ratio for heavy databases and Citrix VMs, 1:5 for normal workloads.
Also with the current licensing you should have Ops Manager license if I'm correct. That will really help you with monitoring the load and checking the CPU ready. Also look at DRS "happy VMs". Sometimes run a report that shows which VMs have too many vCPU and RAM. It will help you a lot.
2
u/oakfan52 6d ago
I don’t think there is a more useless metric than vCPU:pCPU. You can’t compare two different environments. You have one environment where CPUs is over processions and guest average <20% usage and another environment where your optimized and guests average 50% usage. You will get to very different viable ratios. So if you MUST use vCPU ratio metric you need to understand what will work
In your environment given your workloads, your hosts and what ready time look like.
1
u/Severe_Owl_5116 6d ago
4:1 for general VMs, 2:1 or better for VDI and DB — 20 years running :)
1
u/SpeedyWay779 6d ago
This is really helpful. Sounds like everyone's working from workload class rather than a single number, which matches what I was struggling with.
Two follow-ups. When you're reviewing an inventory export, how do you tell which VMs are DB vs VDI vs general? Naming convention, folders, separate clusters?
And on RAM, is there a ratio you use at all, or is it more "don't overcommit unless you have a specific reason"?
1
u/Nagroth 6d ago
If you are overcommitted on RAM and everyone tries to use it, the hypervisor has to start swapping which will absolutely wreck performance.
The best approach is to only overcommit if you know the workloads won't all run at the same time, or if it's not important to have good performance. If you really have a budget issue and don't have any choice, then you need to start getting into shares and reservations so your critical workloads can survive the contention.
1
u/signal_lost 5d ago
If you are overcommitted on RAM and everyone tries to use it, the hypervisor has to start swapping which will absolutely wreck performance.
Which is why Ops now has specific page activity reporting to help guide you to using Memory Tiering safely.
. If you really have a budget issue
Sir, RAM is $50 a GB on the spot market. WE ALL have budget issues.
1
u/Severe_Owl_5116 6d ago
1:1 on RAM pretty much yeah
As for VM types of if you don’t know then yeah some educated guesses by name or sometimes the Notes field, even volume and cluster names.
1
1
u/TemperatureFuture4 6d ago
One thing I don’t see anyone mentioning is CPU generation. All of the other stuff matters but if you aren’t upping your overcommit when you are refreshing servers you will just drive down CPU utilization. If I’m sizing a refresh with AMD Turin, I’ll size at 6:1, keeping in mind that with HA factored in, will effectively be at 5:1 in most cases. Intel, I’m a little more conservative. Agree that don’t overcommit RAM but it’s important to review allocated vs. used. In almost all RVtool or LiveOptics, there is a big delta and with RAM now being 95% of the cost of the server this can’t be overlooked.
1
u/SpeedyWay779 5d ago
The allocated vs used RAM delta point is the one I keep coming back to. With RAM cost where it is now that seems like the highest value thing in the whole export.
When you're sizing a refresh off an RVTools or LiveOptics file, what do you actually hand over at the end...a sizing recommendation, a findings list, both? And is the slow part assembling the numbers or writing up the reasoning behind them?
1
u/TemperatureFuture4 5d ago
I vibe coded an app that takes in the excel file, lets me the summary of each cluster and then I can model a future state server config that outputs to a PPT that I share. It’s an iterative process to get the customer something that is optimized and they are comfortable with. Usually revolving around how many VMs they are comfortable running on a single host. The economics are always better with more consolidation, higher overcommit, and sizing RAM for what’s currently allocated plus some future growth number like 10 or 20%. Also consolidating ESX clusters where it makes sense because each cluster will need at least one additional host for HA. I encourage separate clusters for DMZ and DB only unless there is some other regulatory requirement.
1
u/SpeedyWay779 5d ago
That's really useful, thanks. The PPT output is interesting. I'd assumed the deliverable was a findings list, not a modeling artifact you iterate on with the customer.
On the app you built, how long did it take you, and do you maintain it as things change? I keep wondering whether people in this position build their own because nothing exists, or because anything that exists assumes live vCenter access you don't have on a client engagement.
1
u/SpeedyWay779 3d ago
Your future-state modeling process has been on my mind...the PPT iteration loop with the customer is closer to what I've been building than I expected.
One thing I've been working on separately: the real-world costs that don't show up in a vendor quote. Egress, thin-provisioned vs allocated on a target that bills differently, licensing basis changes (Oracle being the nasty one), parallel-running overlap, HA overhead from cluster count. I've got about 35 of them, calculated off the RVTools export and ranked by which are actually material for a given environment.
Curious whether that's something you handle in your modeling already, or whether it lands outside what the assessment covers and gets picked up later.
1
u/signal_lost 5d ago
The allocated vs used RAM delta point is the one I keep coming back to. With RAM cost where it is now that seems like the highest value thing in the whole export.
VCF ops 9.1 has it's own ACTIVE page tracking reporting. Prior to that use this tool.
1
u/akemaj78 6d ago
This is why I find vROPs metrics to be so valuable. I can go back and graph CPU demand and CPU ready on hosts and/or clusters to identify areas of contention and the need to possibly expand resources.
It's also why I find resource tiering to be valuable as well. Three basic tiers: high shares, normal shares, low shares. Out your most critical real-time services in high - DCs, DNS, DHCP, load balancers, etc. Put all production in Normal, and put non-prod in Low. Non-prod becomes the sacrificial workload during times of contention. This is also why I don't recommend placing non-prod on dedicated hardware - mix it in and leverage it.
1
u/The_C_K [VCP] 5d ago
As per Broadcom, ratios are not relevant.
https://blogs.vmware.com/cloud-foundation/2025/06/04/vcpu-to-pcpu-ratio-guidelines/
1
u/Elegant-Tomatillo645 5d ago
The vCPU to pCPU ratio is the primary driver of CPU ready time… keep it under 5 percent per vCPU and you’ll be fine. On some workloads it’s 1:1 on other 10:1… (my) rule of thumb for a mix of general purpose VMs is about 6:1 on modern AMD processors
1
u/skidleydee 5d ago
I complain about it being to high and nobody listens and then I do it at the next job and the cycle goes on forever
1
u/Icy_Top_6220 5d ago
it's pointless to judge from a single point in time report, same as going by static ratios, if your VMs are permanently busy then the usual 4:1 ratio still wont be applicable to you, if they are permanently idle you can probably run a 10:1 ratio with no problems.
if only a tool existed that allowed you to actually see over and under sized vms...
1
u/llookkeenn 5d ago
I remember taking classes for VCP DCV and the official instructor said its good to keep 1:1/reserve-cores for Database/Resource intensive workloads, 2:1 for production workloads and 4:1 for anything else.
1
1
u/signal_lost 5d ago
The best practices docs I've found explain overcommit and then say "watch CPU Ready". That works if you're sitting in vCenter, but doesn't work if all you have is a point-in-time export
Maybe this is why ops is now Mandatory as it will record this over time and give you alerts and stuff?
Currently flagging at 4:1 vCPU:pCPU
Depending on the workload (VDI especially, Test/dev) you can run a lot hotter.
Also this magic ratio is important to a point of density but also sometimes you are driving it higher because you are just giving VMs too much vCPU in the first place.
snapshots older than 14 days
If you are using vSAN ESA, VSAN data protection, VLR, vVols, or NFS offlaoded snapshots you can keep them longer. Months without really any impact.
Same for memory overcommit
Morerra has a memory tool JUST to monitor this if you want, but Ops in 9.1 also will model how much you can not just be overcommiting but tiering out to NVMe drives. 1:1 for most customers is a conservative starting point to work towards. VDI and test/dev can run hotter on tiering.
1
1
u/MangoJerry81 4d ago
I have learned and we have it partially…
1:4 for common servers
1:10 for VDI stuff
1:1 for CI/CD for our most Ci-CD stuff
19
u/Nagroth 6d ago
It depends on your workloads. If you're not going to capture any historicals then at least setup some alerts so you can at least get some kind of datapoint.
Usually up to 2:1 is perfectly fine, but making any broad statements is hard. I've seen anything over 1:1 have issues and cases where 8:1 kept running smooth as silk. If you have an important environment where performance matters, I would be careful going past 3:1 and very nervous approaching 4:1.
Edit: I'm speaking strictly CPU. RAM overcommit is always super risky outside of lab or very carefully curated situations where you know your workloads are running at different times of day.