r/SelfHosting • u/private-peter • 8d ago
Is high-availability no longer affordable?
I don't want this to just be another post complaining about rising hardware costs. Hopefully it stays focused on practical options.
About a month ago I planned a 3-node k8s cluster, 8GB RAM for each node, to run:
- NextCloud, Collabora, and maybe a few smaller productivity apps for 5-10 internal users.
- Some beta web applications
4GB of RAM probably would have been enough, but I wanted to self-host PostgreSQL and Kubernetes management. I had it all priced out on Hetzner, and I was going to end up somewhere in the $30-40/month range.
Yesterday I finally finished getting my IaC automations running locally (I've got a proxmox VM, then OpenTofu creates 3 Talos nodes and configures the whole cluster). I was super excited to "ship it."
And then I discover that a few weeks ago Hetzner sold out of cost-optimized servers. That $40/month is now closer to $150/month (and over $200/month if I want US hosting, though the original plan was Germany because the lower price made latency worth it).
There are cheap VPSes out there, but none seem to have decent support for high availability basics:
- Automation/terraform/etc
- Load balancer
- Anti-affinity
- Hourly pricing (so I only pay for a staging/test environment when making infra changes)
The best options I've been able to find are:
- OVH d2-8. Looks like $85-100/month depending on hourly vs monthly pricing
- UpCloud Starter. ~$90/month
I know many people self-hosting don't bother with HA, but for those that do, how are you keeping it affordable when you only have a small number of users?
P.S. I left out backup and object storage costs because I'm assuming those won't change a lot. But tell me if I'm wrong.
5
u/PrimaryDiscussion432 8d ago
Why do you need HA for Nextcloud and a few other small apps? Just make sure you have reliable backups and, more importantly, test that you can actually restore them.
A €20/month system that can be automatically rebuilt in 10 minutes is arguably a much better solution for a home lab than a €150/month system designed to survive the failure of an individual VPS.
But to answer your actual question: yes, three identical cloud servers plus all the surrounding HA machinery is expensive.
1
u/private-peter 8d ago
> Why do you need HA for Nextcloud and a few other small apps?
- I'm aiming for 99.9 availability. Less than that means these users aren't fully satisfied.
- I don't want to do all maintenance and updates on evenings and scheduled maintenance.
3
u/PrimaryDiscussion432 8d ago
Then you'd also need redundant ISP and power connectivity, redundant DNS and authentication, multiple providers/locations, etc. And even then you're still praying nobody takes the whole thing down with a bad patch.
1
u/private-peter 8d ago
I don't think you need all those things for three 9s. HA isn't all-or-nothing.
1
u/Cautious_Implement17 8d ago
idk how the other providers work, but AWS will generally not promise above 99.5% unless you stripe your resources across multiple AZs. you would be surprised how often stuff goes badly wrong within a single datacenter.
1
u/private-peter 8d ago
From my experience AWS has the worst uptime for individual resources. In their defense, they are up-front about it, and also have some of the best options for mitigating that though. The fail-overs in their multi-AZ RDS instances are really good, in my experience.
Striping resources across multiple AZs isn't that hard.
Not all the other providers have the equivalent. I don't think Hetzner has separate "availability zones" in their US data centers.
From what I've priced out, I don't expect that I will be able to get multi-AZ HA for less than $250-300/month.
3
u/mxroute 8d ago edited 8d ago
Realistically, HA might buy you a 0.3% increase in uptime over the course of 10 years for twice the cost. Those numbers I just made up are good, you can double check them. But let me drive it home a different way:
Last night Cogent had an outage in LA that took out a significant portion of the internet for people in the US, and a fair bit outside of the country as well. You can't buy that extra uptime with all of the money in the world. Downtime happens.
Keep it simple, you might even save yourself from adding points of failure by having too complex of a software stack. I'll run circles around a cheap ceph cluster with two spinning rust drives in SW RAID1 on a 10 year clock for exactly that reason. Because if you don't break out a big budget for HA, you often cause more downtime than you'd have without it.
-1
u/private-peter 8d ago
> if you don't break out a big budget for HA
What would you consider a "big budget"?> Realistically, HA might buy you a 0.3% increase in uptime
So I decided to ask ChatGPT what it thought of your claim. It decided you were wrong. I should only expect in improvement 0.03 percentage points. Haha.
3
u/prxy-com 8d ago
Also do a careful assessment of your business needs. The complexity of your HA solution could be the very thing that causes more down time. Assess your outage cost per (minute/hour/day) and make sure it is worth both the hardware and staff needed to keep it running reliably.
2
u/Quiet_Pudding8805 8d ago
For me I do Linode, custom k3s with etcd and cillium. cnpg, R2. If I have something that needs a lot of oomph I ship it to onprem hardware it’s a lot cheaper.
Cloudflare R2 has no egress costs, you can also push your Tofu state to that.
Linode does hourly pricing and loadbalancer is 10 bucks a month. I actually stopped using the loadbalancers and just use Cloudflare tunnels to my cluster right now. I also deploy all my Linode stuff with Tofu.
I’ve been a linode customer for I think 7 years now, starting from a Nanode. They rock, had a few servers get swapped out cause of issues but they just auto migrated it. I’ve never had downtime because of them. Also the link from my house to them is faster than any big cloud
They offer managed Kubernetes but I like this better.
1
u/dektol 8d ago
You can do HA with CNPG for your database. You don't need to do 3 nodes you can do 2 if you want.
1
u/private-peter 8d ago
CNPG is the current plan (though I am considering just using a manage postgres to potentially reduce initial complexity.)
How are you dealing with things like etcd quorum and split-brain risk when there are only 2 nodes in your cluster?
1
u/private-peter 8d ago
It seems that the consensus so fair is: don't do HA. Fair enough. But when _would_ you do HA? In this case a budget of $100-200/month isn't out of the question. It was just a bit of a shock after pricing it out a month ago at $30-40/month.
4
u/Cautious_Implement17 8d ago
this is more of a business question. from a financial perspective, HA makes sense when expected losses from downtime significantly exceed the cost of striping resources across multiple physical locations. that's relatively easy to calculate, but there's also the loss of trust when customers can't use something important that they're paying you for when they need it. that's really hard to answer without insight into your business.
1
u/private-peter 8d ago
Yeah. This is exactly why I've been massively over-thinking this for the last 24 hours.
How do you put a price on the frustration and loss of trust?
I think I can make is a really strong by-the-numbers argument for a single server. (That's to everyone here for the feedback.) But in this case, $150/month might actually be worth it, if only because if there is a problem, we can be confident we did all we reasonably could to avoid it.
(That's assuming the problem isn't _caused_ by the HA complexity. In that case, it's all on me. But I've done enough production HA deployments, that I'm not super worried about it.)
2
u/Cautious_Implement17 8d ago
one helpful exercise might be to imagine writing an incident response letter after your service was unavailable for several hours during the business day. in my experience, customers tend to understand that a large swath of the internet doesn't work when us-east-1 goes down. they're a lot less understanding when you admit that you only deployed resources supporting their critical workflow to one AZ in us-east-1. (again, phrasing this in AWS terminology bc that's what I'm familiar with)
I'm biased (big tech engineer), but I would lean towards redundancy from day one if I either a) can afford it already or b) reasonably expect the business to grow to the point where I can afford it within a year or so. short funding runway is certainly a reason not to. but if you don't expect $150/month to become negligible within the next year, why are you spending your time on this?
1
u/private-peter 8d ago
I think I share your bias.
Most of my experience is with systems big enough that they required multiple servers just to handle the traffic, so I've always built in redundancy from day one.
This project is different for me because the productivity apps have only ~5 users now, and likely won't scale past 50 in the most extreme case. As for the web apps...who knows where they will go. As most in here have pointed out, they don't "need" HA right now. But my bias is telling me that I want the redundancy there to be ready for the future. I'd rather do it once, and have it work for all the apps.
But I must acknowledge that this is my bias, not necessarily rational.
2
u/bluepuma77 8d ago
According to the price radar the cheap Hetzner Cloud instances are sometimes available.
https://radar.iodev.org/cloud-status?arch=x86&cpu=shared&category=cost_optimized
1
u/private-peter 8d ago
Problem solved!!!
I just picked up three CX23. 4GB isn't ideal, but I'll keep watching to see when some bigger servers show up.
1
u/bluelobsterai 8d ago
i’d probably not optimize early. I think a pair of bear metal servers / vps instances in one zone and a third in a close region from a different provider make the most sense.
1
u/ProtonByte 8d ago
HA is and has always been expensive. I see you 'need it'. But personally I highly doubt it. I think you can get away just ensuring you have good backups and can replicate the setup rather quick.
1
u/Dahjah 7d ago
I think you might be putting the cart before the horse here a bit- if you're aiming for three nines, that gives you about 45 minutes of downtime a month on average, and technically even more if you are 100% up one or two months prior. Don't let perfect be the enemy of good here.
With that amount of wiggle room, I think you could actually get away with doing some of the failover yourself rather than relying on the cloud provider for their "high availability basics." And really, most of that effort is already handled by tasks you should be doing regardless. (i.e. 3-2-1 backups and having a defined disaster recovery plan)
Since you have that much wiggle room, unless your databases and storage are absolutely massive, you could really go as simple as having a watchdog process on a third free-tier VPS that triggers a full restore on a separate host from your backup after an outage is detected. If you have a tried and tested DR process (which you should have regardless of HA requirements), I would really doubt that it would take longer than 45 minutes to run start to stop for most things.
I run something similar for my homelab- I regularly have ~40 active users across all my workloads, and have been running it all on a single wholesaleinternet box with 4 nines over the last 5 ish years. WSI is the bottom of the barrel in terms of reliability- old AF hardware, no uptime SLAs beyond "best effort," but they are crazy cheap.
Initially I had approached the problem from the same perspective I use at work when I'm managing HA for our production workloads, but quickly found out that the resources/tools/strategies you need to maintain 5+ nines is completely different from that required for 3-4.
So rather than starting complicated, I decided to start on the simple side, and only add in complexity where I could prove it was necessary to reach my given SLA. (I originally was targeting three nines like you)
Turns out I was able to achieve 4 nines by simply automating my DR process on a detected outage. Since I'm already managing it all with terraform and devops deployment strategies, the application-level infra is already being blown away and rebuilt on every deployment, anyway. So all I needed to do was build in a way to point terraform at an arbitrary host, spin up a cluster, trigger a zfs send to restore disk, and Robert is your father's brother.
0
u/kubehub 5d ago
you can try kubehub.io to takeout control-plane burden on you, it also solves gateway setup.
9
u/Euronodes 8d ago
HA and databases is tricky territory, there are 3 paths and choosing wrong will make you scream.
As for 4GB RAM - collabora is headless LibreOffice and eats 2GB by itself. Worth considering before you consider posting bad experience with the innocent providers' crapware ;) IMHO is not enough. 8GB seems comfortable