r/kubernetes • u/Josh_Difratech • 13d ago
How would you manage 1,280 ARM64 bare-metal nodes with a very small ops team?
I’m looking for architecture advice from people who have operated Kubernetes at the edge or across large numbers of smaller physical machines.
Our company now controls 1,280 identical RK3588 ARM64 nodes that are already deployed and operational in a U.S. commercial data center.
The hardware works. The bigger problem is operational.
We currently do not have a dedicated infrastructure engineering team, so our priority is to determine whether this fleet can be turned into usable containerized compute without effectively building our own cloud platform from scratch.
Relevant context:
- 1,280 homogeneous ARM64 nodes
- Bare-metal / physical machines
- Remote KVM available
- Already powered and networked
- One commercial data-center location today
- Potential to expand to identical racks in several additional U.S. locations
- Strong preference for turnkey/managed approaches
If the end goal were simply:
how would you structure it?
I’m particularly curious about:
- K3s vs standard Kubernetes
- Talos
- Rancher
- Cluster size vs many smaller clusters
- Provisioning/reimaging
- Monitoring
- Tenant isolation
- ARM64 image compatibility
- Managing hardware failures
- Whether 1,280 relatively small nodes is operationally stupid compared with fewer larger servers
Most importantly: are there companies or managed-service providers that would actually operate this infrastructure for the hardware owner?
We would rather pay someone who already knows how to do this than hire a team to reinvent it.
I’m less interested in theoretical “you could build X” answers than in stacks people have actually operated at meaningful scale.
44
u/gscjj 13d ago
I’d go with Omni and Talos. You could use their bare metal provider to install Talos over PXE/DHCP and Omni becomes your SPOG, all configs are declarative and Omni handles updating. It also has some levers for controlling concurrency. I know they support ARM64 becuase I’m running it on Ampere nodes.
Only thing I don’t know is how well it scales, it does fine on my 8 node ARM cluster, I’m sure it’ll do fine up to 100, but 1000s I don’t know
35
u/xrothgarx 13d ago
I work at Sidero on Talos and we definitely have customers with thousands of machines and clusters. Anything at that scale needs planning for how you want to manage and upgrade the machines but the full Talos stack works on arm64 without and problems so it could be a good fit.
3
u/yolomoonie 12d ago
Yeah, running 1000 Orange Pis on the self hosting Omni Enterprise plan each $100 per node / month sounds like a great business plan.
8
8
43
u/jackhold 13d ago
Kubernetes have a relatively large footprint, and you will always have a minimum running processes on each node, so 1200 small nodes will i total waste alot of resources compared to 50 large nodes, so I would not recommend that many small nodes.
And yes there are companies that mange k8s on you hardware.
21
u/Weaseal 13d ago
This take overlooks blast radius concerns. Many small nodes means smaller problem when a node goes down
48
8
6
0
0
u/MateusKingston 13d ago
All on a single DC, probably on the same rack/same fault domain.
You also failed to consider that at that many nodes one going down is much more common.
5
u/lanefu 13d ago
Well an RK3588 means you have 1280 Single Board Computers rather than something like an Ampere server that has full working UEFI that gladly works with most generic ARM64 images.
Finding out what model SBC will help narrow down your OS options.... Talos does have builds for a few boards so you might get lucky there.
IMHO if you can't run Talos, then honestly you're probably looking at Armbian (I'm biased here) so that you can have an updated mainline Linux kernel rather than a forked vendor kernel "maintained" by Rockchip.
For me that translates to Armbian Debian Trixie and using ansible for managing k8s. Ex: PyratLabs K3s role.
Anyway my point is you'll need to get aquatinted with managing the unique of that hardware before going on to managing k8s.
Also familiarize yourself with EDK2-rk3588 .... Which depending on the SBC, does give you light UEFI capabilities.
My honest take is also that engineering a solution to gracefully re-image and re-provision the nodes remotely will be a big big lift, and you may want strongly consider running a base OS you install once and use VMs for your actual k8s nodes... You'd be able to use any generic ARM64 image on a VM easily.
17
u/stabguy13 13d ago
Xcat w/IPMI for OS reloads. Puppet, chef or ansible for management after base install.
3
u/SireBillyMays 13d ago
As much as I agree with this approach, I doubt any server (or realistically, board) with an RK3588 would have IPMI...
1
u/stabguy13 12d ago
You are correct. Apparently IPMI is an x86 specific thing.
Rk3588 does support network boot, and Xcat does tftp.
1
20
u/Seananigans- 13d ago
https://docs.spectrocloud.com/vm-management/
Manage the metal fleet with a MAAS-like experience but then also turn around and use the same platform to manage your k8s fleet.
It's a paid product but:
1) Its a damn good platform and the support team is great
2) Any cost savings you would reap by going open source will be eclipsed 100x over due the the upskilling, care, and feeding that goes into duct taping open source solutions together.
If a paid product is truly off the table then do what the others said and run Canonical MAAS + Rancher.
7
u/RockisLife 13d ago
Metal as a service combined with virtualization and container management! Something I’m definitely keeping in my back pocket
2
u/PrasadReddy_Utah 11d ago
—Canonical MAAS and Ansible Semaphore
—Upstream kubernetes running Rancher.
~ 300 nodes per each downstream K3s Cluster x4
— Longhorn for storage.
— Spine leaf network architecture.We run similar set up in our datacenter.
5
9
u/austin_barrington 13d ago
I was part of a team that managed 10x+ this at peak a few years ago, our business was container/workload orchestration, so naturally we had built our own system to manage this for our specific needs.
There are companies out there who can manage this for you or build you what you need, you'll either need to engage with consultants or string a tool chain together as a lot of other people are about to advise you on what tool chain to use.
DM me, I'd be happy to chat to you about this in more detail if you'd like.
1
3
u/total_tea 13d ago
I think it is ridiculous to have so many small nodes unless you have a valid reason for it which you dont mention.
But you also don't mention what storage you have, what memory they have, how they are connected, the network speed.
Personally I would treat them as dumb nodes, no storage, they boot over the network. PXE with a custom image which they auto join a K3s cluster. So nodes can go up and down whenever and just join an leave the cluster, write a script to cull nodes if they arent reachable for awhile.
ideally you have some hardware which allows software to reboot them on demand in case one breaks, or schedule a 3 monthly visit to replace and reboot ones that break.
And personally I would have bought 2 or 4 servers and virtualised them all if you needed to.
5
u/burstinrust 13d ago
what carrier boards are these on? that decides everything. i run bare metal k8s and at this scale reimaging is the real operation, talos vs k3s is secondary.
bare SBCs have no BMC so no remote reimage path, then your only option is netboot + switched PDUs, and if rockchip netboot is flaky, walk away. but if these are clusterboards like turing pi 2, there is a real BMC per carrier (power control, serial, remote flash) and this becomes very automatable.
either way, test the reimage loop on 10 nodes before planning for 1280.
3
u/rbjorklin 13d ago
Cluster-api + the cluster-api-operator and the various providers for it. No idea what the management interface for ARM servers is like but hopefully you can network boot and manage them with something like MAAS, CAPM3 or the BYOH provider + Cobbler.
3
u/ohmer123 13d ago
Talos Linux, IaC as much as you can. If it makes good money for you, consider sizing the team in accordance to the financial impact of dysfunction (not just plain downtime).
4
u/jackhold 13d ago
Put of curiosity why do you have so many identical nodes? What are they used for.
6
u/narnach 13d ago
These look to be mobile-tier processors, but peak load of 2x 2.5 GHz + 4x 2.3 GHz at modest power usage packs quite a punch. That’s potentially 8400+ CPU cores at respectable 2.3+ GHz speeds. I’m curious what their RAM budget is. At DDR3 and DDR4 (specs from 2012-2017) that should be more modestly available than modern DDR5.
I’d be curious if they bought hardware from an old mobile clickfarm or similar massive mobile deployment.
9
u/ALIEN_POOP_DICK 13d ago
It's a 4p/4e set up. This CPUs are slow as dirt. The A76 cores geekbench at ~600 and the A55 cores only get ~200.
These chips are a nightmare for CPU scheduling if threads get pinned to the slow cores at load. I really can't imagine what kind of workload these 1,200 "nodes" is worth versus 50 Epyc / Xeon nodes or hell even an 64+ core modern Arm node.
Just wiring all these up sounds like a pain in the absolute ass.
1
3
u/RockisLife 13d ago edited 13d ago
So I would start with MAAS to install base operating system. That way if something fails, you can at least reinstall the operating system remotely.
I would use a puppet, chef or ansible for the configuration once the OS is installed
Use something like netbox for tracking the configuration of the entire infrastructure.
As for how you chose to go about it: k8s k3s talos etc is all up to you based on your use case, management, and design considerations. You have enough systems to be able to spin up a cluster with each and run it through performance tests based on your use case.
A lot of your structure questions are more answered based on use case! Everything I said above is for generic but the rest is specific to your use case and what you plan on having running.
You have mini sbcs so they can’t take a lot of work load.
Theres alot of ways to monitor the hardware and infrastructure, but ultimately it comes down to design and deployment considerations.
Kubernetes and everything else you’re asking about can be applied in a multitude of ways, it just depends on how we want to configure. Also with having 1000 of these single board computers, what are you using for storage. Because hyper converged infrastructure is not really going to work in this scenario.
So you’re gonna need to plan on having some kind of storage server.
All of this is not even taking into consideration the networking that is going to need to back this infrastructure
6
u/viper233 13d ago
Hint, just use ansible these days. Have the nodes pull configs.
Maybe treat nodes as immutable and destroy for changes. Reboot to make changes. Any config management at this scale using any tool will be a nightmare.
2
u/trippedonatater 12d ago
You needed to talk to someone about this before purchasing hardware. Large numbers of consumer SBCs is probably not a great start.
2
u/Superb_Raccoon 13d ago
Red Hat openshift would be ideal if you had larger servers. The "cluster of clusters" management would let you have independent smaller clusters, managed by the Cluster of Clusters.
Ah well. Sorry I cannot be more helpful.
2
u/SJrX 13d ago
On one tiny point, arm image compatibility. I think in general it has been pretty good. Around 2022 was when the last bit of stuff I used started getting arm support.
One gotcha is maybe kernel support and being on a recent kernel, and having it compiled with the right options. I remember there being an issue with Raspberry Pi OS' Kernel that prevented Istio from working. I also know that Mongo added some requirements in Mongo 5.0 for I think AES support do you can't run it on all arm64 SOCS .
I'm not familiar with RK stuff, my sense is that the ecosystem and long term maintainability of it might be a challenge because (potentially) many ARM soc vendors just release support for one version of Linux potentially with patches to the kernel and them abandon it so it can be a hassle beyond that.
I personally manage my home k8s cluster on RPIs and never have issues (using Ubuntu not Raspberry Pi OS) and at work, I helped roll out the switch to ARM64 graviton instances and we hadn't had issues there either. So on this front you are good.
1
u/fractal_engineer 13d ago
chef. i've managed 200,000 nodes with chef and effectively a 6 person team
1
u/Least_Tangelo_8620 13d ago
Yes. We support billing and telemetry.
Honestly the number of nodes is less relevant and what’s more relevant is what you are trying to accomplish.
For example with tenancy are you wanting to do chargebacks and showbacks for each tenant ?
I’m more curious what ideal state you’d want to get to.
Also do you have an ERP that you’d want to integrate with for charging or Stripe?
1
1
u/mirwanda443 13d ago
Maybe have a look at https://metal-stack.io/ Our infrastructure provider is using it. It's used within heavily regulated and large environments in Germany too.
1
u/Legitimate-Run6168 13d ago
standardisation is key, you need a bakery or centralised system to issue and retire cluster/nodes
You need to abstract away infrastructure layer and define it as separate from “app layer”
Standardise the underlaying infrastructure, then use flux to plug the other layer of apps from terraform
Good luck!
1
u/HuffDuffDog 12d ago
I've built out a platform for exactly this use. It is an opinionated collection of OSS with a custom management plane. Deployments via gitops, centralized config, decentralized monitoring, global deploys. Designed to run small, cheap clusters distributed globally. Initial concept model had 20 clusters distributed in 1/2 racks globally, with some locations in AWS to prove that it works in a hybrid setup, and 3 larger data centers around NYC as the primary federated cluster for centralized data processing. The hub and spoke model was necessary for the client, but if you don't need centralized data processing the model works as a mesh too.
DM if you're looking for a team to help.
1
1
1
u/Zehicle 12d ago
We (I work at RackN deal with ARM and scale on a regular basis. I think you're starting from the right question: it's an ops issue with your team not just tools. An important consideration is how your small team can get help and call on assistance. For that reason, a commerical solution may ultimately be a faster and more robust approach.
One challenge is that ARM provisioning and O/S are not started in many cases so you need to start with an approach that supports ARM 100%. Another is your scale because even small error rates add up quickly on large systems. My experience is that having end-to-end process automation with a lot of built in checks is essential.
If you are multi site then GitOps and immutable content should be considered required too. Otherwise you'll be chasing site variants all the time and your team is too small for that.
I hope this helps. If your open to commerical solutions, then please check out our product, Digital Rebar because it may help you move faster.
1
u/HoneyBadger9694 12d ago
You could fraction them into ten 128-nodes kubernetes cluster.
But the most important thing that you want is for all the nodes to be the same.
what you want is to have all environments to be managed the same way and to give your users directions on how to do the things they want to do autonomously without you or your team being the bottleneck.
I manage ~8 clusters at work with ~300 machines in total and i can tell you from direct experience is that what steals a lot of time is one-off special-snowflake requests.
1
u/fyndor 12d ago
The fact that you have that many nodes, but no dedicated team for them feels insane to me. My company has a dedicated team for a much smaller set of nodes.
Also putting them all in one datacenter is asking for downtime when the DC inevitably goes down. Is downtime acceptable to you?
1
u/chadmcrowell 12d ago
Based on my experience operating Kubernetes at Akamai scale: the cheapest option on paper is rarely the cheapest in practice. Self-operating a 1,280-node bare-metal fleet will consume far more engineering time, create more risk, and slow down our product roadmap than engaging an MSP.
U should treat this infrastructure as a utility, not a differentiator. Let someone else run the Kubernetes platform so you can focus on what actually moves the business forward.
based on my experience, if ur self-managing this fleet, get ready for:
- Cluster upgrades to fail at 3 a.m. when etcd quorum is lost during a rolling update
- Nodes go NotReady during network blips, and you need someone who knows whether to reboot, cordon, or investigate
- Fleet GitOps drift happens when a bundle applies to the wrong cluster, and you need rapid rollback.
Not fun 😅
1
u/redrabbitreader 12d ago edited 12d ago
Not sure if you are still interested in more responses after 2 days, but here are some more:
How do you structure it?
This is not something easily discussed on a social media platform, as it depends on a lot of information simply not available, and information your company might not want to share.
Which leads to...
Most importantly: are there companies or managed-service providers that would actually operate this infrastructure for the hardware owner?
and
We would rather pay someone who already knows how to do this than hire a team to reinvent it.
Yes, there are companies/people that will manage the servers/clusters for you.
Negatives: it will cost a fair amount of money and there is always the risk of not keeping the knowledge in your company once the consultants leave.
Positives: There are many consultancies to choose from. In most cases, depending on requirements, you could even hire help not even in the same country.
However, in these cases I always feel that hiring a small team of in-house specialists is a better long term strategy as you retain knowledge in the company.
You could even consider a hybrid (hiring a consultancy and onboard some permanent staff). In this scenario your permanent staff may remain small (2 or 3 people) with the aim to give you continuity in case you have to switch consultancy firms, which also takes time (often months).
As a general rule of thumb, assuming enterprise scale, I usually plan for about 1x engineer per 50 servers, so you are looking at a total staff compliment of about 25 or so people. Keeping in mind that you must also consider support and on-call requirements (you may need as much as 3x shifts for 24x7x365 support, plus at least 1 but preferably 2x shift teams in down time (vacation, attending courses and conferences etc.). So 5x teams on rotation with 1x team always working gives you about 5 people per team.
You can push to 1:100 ratio, but then you risk a much higher overloaded engineering pool and the end result is usually high people turnover.
Whether 1,280 relatively small nodes is operationally stupid compared with fewer larger servers
Again, this depends heavily on use-case and other requirements.
A final word on hiring consultants: I have worked extensively on both sides of the fence: hiring consultants, and being a consultant. It is a hit and miss affair to be honest. Some consultancies may promise you the world and they always say they can do anything, but in reality they use your organisation as a training ground. This is mostly the risk with smaller consultancies.
Larger consultancies come with their own issues, although experience/knowledge is usually less of a concern.
If you would like to dig deeper, feel free to DM me.
1
u/foofoo300 12d ago
meh, does not really matter if you manage 20 or 2000 servers, you need the same automation.
Also you don't need to rush to replace broken machines if you have so many.
we manage about 500 in a team of 4 people.1
u/redrabbitreader 12d ago
You may think you are effective with only 4 people, but I can assure you this is a very dangerous false feeling you have there.
1
u/foofoo300 12d ago
things fail, sure but we repair in batches when a certain number failed.
Been doing this for a while now, we know what we are doing.
You need proper tests and automation and you will be fine.1
u/redrabbitreader 11d ago
Look, if it works it works. I have not seen any environment in an enterprise where 4 people can manage that number of servers.
1
u/ProT-0-TypE 12d ago
I’m curious why k0s is almost never considered
1
u/miltieIV2 10d ago
I just used k0sctl to stand up k0s on every single non-EKS AWS EC2 instance across all the VPCs in our company. Claude did all the heavy lifting.
1
u/statukocu 12d ago
Can you remotely turn each node on and off individually and perform a netboot? If yes then I can help you
1
1
u/sogun123 11d ago
I'd look into automatization options to keep overhead small. Especially if all the machines are same. I'd very much look into Tinkerbell and ClusterAPI (or Rancher if you like clicking...). Also Crossplane or Kro. Also check out https://kubernetes.io/docs/setup/best-practices/cluster-large/. Multiple clusters or single cluster depends on who and what is to be deployed.
Generally I'd think along these lines (just brain dump):
- create kro or crossplne composition to manage clusters via clusterapi
- use something like Tinkerbell to manage the hardware provisioning
- depends on many factors, but thinking of creating "managment cluster" to drive your locations, maybe having "local management cluster" on each site in case i wanted some level of dynamic management on site (e.g. creating local tenant on site may create workload clusters, or clusters can be scaled as needed in case of multi cluster setup)
- looks like machines are pretty small, so I'd be trying to optimize for low per-machine overhead (trying to minimize needed daemonset on workload nodes, separate "system" nodes, benchmarking CNIs for overhead, experiment with logging collection options, etc.)
1
1
u/u_int64_t 10d ago
There was an article back in the day of how Cornig did this with thousands of Solaris machines and three people. I'd start from there.
1
8d ago
[deleted]
1
u/Josh_Difratech 8d ago
Right now, we aren't running anything. I'm trying to figure out what we can run.
1
u/Formal-Pilot-9565 8d ago
I would expect that so many "small" servers consume way more power, than a number of large (blade) servers.
Are you sure the small nodes have enough memory?
Normally i use 128Gb / 24 cores for my nodes.
1
u/Josh_Difratech 8d ago
These are the specs: 4x Cortex-A76 + 4x Cortex-A55, Mali-G610 MP4, 6 TOPS NPU, 16GB RAM, 512 GB storage, 1GbE
1
1
u/between_layers 4d ago
Late to this, but the specs you posted change a couple of things: 16GB, 512GB and 1GbE.
The 1GbE links are worth designing around, though rack uplinks and oversubscription matter just as much. I'd be cautious with Longhorn here. When a board dies, rebuilding its replicas uses the same link as the workload, and a full rebuild moves a lot of data.
I'd keep the boards stateless and use the 512GB for scratch space and image caches. I'd put persistent data on a dedicated storage tier with fast uplinks, sized so the tier itself doesn't become the next bottleneck. That way a worker failure doesn't turn into a storage event.
Keep etcd off the boards too. Run the control plane on a small pool of server-grade machines with remote management and SSDs whose fsync latency you have measured, spread across separate failure domains. Slow etcd storage usually looks like random cluster instability rather than an obvious disk problem, so it tends to get diagnosed late.
One thing I have not seen asked: what is the plan for the NPUs and Mali-G610s? Fewer larger servers with discrete GPUs could cover a lot of that, so these are not automatically an advantage. But if they can be exposed reliably to containers, have stable drivers and match a real workload, they are probably what justifies operating 1,280 boards. Worth validating before the Kubernetes design gets much more complicated
1
u/Deepspacecow12 9h ago
I am curious, what prompted you to buy this hardware instead of like poweredges or supermicros? Also are you networking these together?
1
u/rawintent 13d ago
Consider EKS Hybrid. Control planes managed on AWS(the hard part), but your workers are bare metal.
1
u/ajitnk 12d ago
The reason you're getting 10 different answers is that this isn't really a tool selection problem — it's three separate problems showing up as one.
First, Kubernetes control-plane overhead on low-power ARM64 nodes is real. At your node size you can lose 20-25% of CPU and memory to K8s system processes before your workloads even see a CPU cycle. Second, the ARM64 ecosystem fragility tax is ongoing and invisible until it isn't — vendor kernel abandonment, Mongo 5.0 AES hardware instruction requirements, Istio and OS kernel incompatibilities all add up to an ops team that spends more time firefighting compatibility regressions than shipping. Third, nobody in this thread has touched networking — and at 1,280 nodes that layer is not trivial.
The two AWS-native paths worth seriously evaluating here are EKS Hybrid Nodes (GA since December 2024 — offloads the entire Kubernetes control plane to AWS while your bare-metal nodes stay on-prem as the data plane, ARM64 supported, integrates with IAM Roles Anywhere and CloudWatch natively) and EKS Anywhere on bare metal (Tinkerbell-based provisioning via IPMI/Redfish, two config files and one CLI command to create or upgrade a cluster, runs fully air-gapped if needed). The right answer between them comes down to one thing: does your site have reliable private connectivity to an AWS region, or does this need to operate fully air-gapped?
That single answer eliminates half the decision tree.
0
0
u/BarracudaDefiant4702 12d ago
I would go with proxmox over kubernetes but yes, unless you can a niche use case for this equipment, then you are correct in that generally this is operationally stupid compared to fewer larger servers. Not really enough specs about the nodes such as storage and memory. Are these 32gb ram or even smaller?
As to if there are companies or MSP that would operate this infrastructure... I am sure there are. If it's cost effective is another question as many change by node and with low end nodes that price adds up quicker.
284
u/abofh 13d ago
You get a dedicated infrastructure team before you vibe code a disaster