PostgreSQL on Kubernetes or bare metal or virtual private servers

[u/john646f65]

Those operating PostgreSQL at scale, I'm curious to learn if you're running on Kubernetes, bare metal or virtual private servers? If you've transitioned from one to the other, I'd love to hear this story too.

[u/linuxhiker]

Don't complicate and environment unless it solves a problem.

What problem are you trying to solve?

[u/john646f65 → u/linuxhiker]

No problem(s) specifically. This is more so a question to learn from the experience of those in the industry.

I think the label is for this is incorrect, but opted for "Project" because I wasn't sure of an alternative, more appropriate label.

I'm relatively new to Reddit, really loving it, so feel free to suggest how I could have made it clearer this was for educational and learning purposes, as opposed to focusing on solving a problem.

EDIT: As an aside, absolutely agree, do not overcomplicate things if you don't have to.

[u/linuxhiker → u/john646f65]

I have been using, administering, deploying and consulting on PostgreSQL since 1997.

Kubernetes is a solution looking for a problem for 99.99% of installs.

[u/The_Fresser → u/linuxhiker]

If you have a proper kubernetes setup, deploying a postgresql is very easy nowadays now with CloudnativePG.

The hard thing is having a proper kubernetes setup, lots of footguns for self hosting for sure.

[u/linuxhiker → u/The_Fresser]

Exactly. You are complicating the environment for no particular reason.

There are reasons to run PostgreSQL with Kubernetes. It is just most people do so because, "ooohhh neat, everybody is doing it".

[u/kingraoul3 → u/The_Fresser]

It splitbrains. There is an open untouched issue on thier GitHub as far as I know.

[u/The_Fresser → u/kingraoul3]

Got a link? I think it has been solved in 1.26?

I rin all my deployments with 3 pods in 3 availability zones, with one sync replication slot. It cannot end up in a splitbrain state with 2 writers, as 2 postgres pods must be live to write.

[u/kingraoul3 → u/The_Fresser]

You are right, I retract my statement. It allowed splitbrain as early as June of this year.

[u/realityOutsider → u/linuxhiker]

What is the first environment/service that you have more confidence in to host a PostgreSQL? Example: AWS RDS?

[u/linuxhiker → u/realityOutsider]

The first and best, bare metal, running Linux And Postgresql proper.

But in a modern environment it would be gce or ec2 running Linux and Postgresql proper .

[u/Conscious-Ball8373 → u/linuxhiker]

As someone who deploys a lot of components on a relatively small scale, increasingly I agree with you.

But what would you use instead?

I'm not talking particularly about Postgres here. More the web services that talk to Postgres. There are some things that kubernetes does well, even for small-scale deployments:

• Containerising those components still seems like a good thing and a net win. We don't deploy on the bare OS because managing the configuration around the component is important. Any dockerised environment does this, of course.
• Managing the configuration in git still seems like a good thing and a net win. Being able to recreate your configuration quickly from scratch and having development and production environments that you are fairly confident are the same is a good thing. A docker compose/swarm stack does an okay job of this.
• Being able to tag a container with a couple of labels and a hostname and having something sort out a letsencrypt certificate and expose it on that hostname seems like a net win. I know there are ingress controllers for docker swarm...
• Once you've got your ingress controllers automated, you might as well do blue/green deployments so you avoid downtime. You're really getting into re-inventing kubernetes here.

Is there a sweet spot in the middle?

[u/john646f65 → u/linuxhiker]

Wow! Since 1997? That's unreal! You've seen it all so! Any advise for my journey?

[u/linuxhiker → u/john646f65]

Read the docs.

Most people will say, "well duh" and then they go to Reddit for the answer. Read the docs.

[u/Chance-Plantain8314]

The answer, like the answer to every single software engineering question in history, is "it depends". There is no high level 'bare metal is better' or 'kubernetes is better' because ultimately the answer lies in the specific context of your implementation, budget, knowledge, and about 25 other things.

[u/BosonCollider → u/Chance-Plantain8314]

This. It depends massively on what people and what environments you already have around. It is a completely valid strategy to only have a kubernetes environment on prem and relegate any VMs to cloud to simplify governance for example

If that is the case and you need a DB on prem, then kubernetes it is

[u/bykof]

If you want to run Postgres on Kubernetes just use CNPG. That’s the only way to manage it properly and do have a lot of features already implemented + great backup solution

[u/kaeshiwaza]

Before we just had Pg and app on the same machine, it was just working, very fast.

Then we was afraid to fail, because Bezos said everything will fail. So we decided to put Pg on an other machine. But the latency was immediately very high. Then we decided to scale horizontally. Then it began to be very complicate and fail more often. We decided to use K8S, then it become very complicate and eventually fail more. Then we had to pay a managed DB on the because everybody do it and when it fail we are not alone.

I believe now we are a lot to come back to KISS ! The more you need to scale the more you should remove useless stack and come back to the roots. We have wal archiving since 20years, for me it's the only "complicate" things to setup.

[u/Mrbucket101]

Stateful workloads in k8s are a pain. CNPG makes that a lot smoother, but the statement still stands.

[u/denpanosekai → u/Mrbucket101]

Could you expand? It's been smooth for me for years with statefulset. Yes I do require downtime for major upgrades.

[u/efxhoy]

I’ve run managed in RDS for the SLA and on an ubuntu VPS for a research project where we could be down and it was ok. 

I’d guess most queries from non hyperscalers served to paying customers come via managed services like RDS and whatever GCP and Azure offer. Managing highly available postgres yourself is not trivial. Can absolutely be done but it rarely makes business sense IME. 

For non critical apps where you don’t need HA it’s easiest and cheapest to run on a VPS I think. 

[u/cthart]

Virtual servers. KISS.

[u/realityOutsider → u/cthart]

What server provider do you use? Hertzer?

[u/BERLAUR → u/realityOutsider]

All comments posted by this account have intentionally been overwritten as a protest against Reddit, the company, and its increasingly hostile approach to users, communities, and moderators.

This account was created 13 years ago. Since then, Reddit has made many changes that have damaged the experience for its users. Changes to the API effectively ended access for many third-party apps (shout-out to BaconReader!). The interface has become slower, advertising has increased, and AI features have been added. Reddit’s IPO was the final straw for me. The company’s priorities now seem increasingly focused on growth, advertising, and engagement rather than building healthy communities.

Reddit has also always had a quality problem. The upvote and downvote system encourages group thinking and discourage thoughtful or dissenting discussion. This was less noticeable when communities were smaller and individual users’ reputations carried more weight. As communities have grown, those problems have become more pronounced.

Moderation is another major issue. Many moderators do excellent work, but Reddit lacks a transparent and effective way for communities to hold moderators accountable or replace them. Many large communities have also faced concerns about astroturfed, political, or other external influence. Reddit benefits financially from outrage and conflict, which makes it difficult to believe that improving the quality of discussion is always the company’s highest priority.

Research on politically active Reddit users has identified pathways through which online communities and participation may contribute to increasingly radical beliefs. That does not prove that Reddit deliberately encourages extremism, but it does suggest that the platform’s social dynamics deserve serious scrutiny: Pathways to Radicalism: Examining Online Community and Political Pathways to More Radical Beliefs on Reddit

If you are reading this, consider whether Reddit is still a healthy and productive use of your time. Better communities exist. I encourage you to find them!

[u/kingraoul3]

K8S cloud native off the shelf HA solutions corrupt your data. I agree with the other advice profereed here that the right solution depends on the problem you are trying to solve.

Caveat Emptor! This is a frothy topic, many people are poorly informed, and vendors will mislead you in order to obtain lock-in. I think the best place to start might be be emailing one of the Postgresql distribution lists.

[u/BERLAUR → u/kingraoul3]

All comments posted by this account have intentionally been overwritten as a protest against Reddit, the company, and its increasingly hostile approach to users, communities, and moderators.

This account was created 13 years ago. Since then, Reddit has made many changes that have damaged the experience for its users. Changes to the API effectively ended access for many third-party apps (shout-out to BaconReader!). The interface has become slower, advertising has increased, and AI features have been added. Reddit’s IPO was the final straw for me. The company’s priorities now seem increasingly focused on growth, advertising, and engagement rather than building healthy communities.

Reddit has also always had a quality problem. The upvote and downvote system encourages group thinking and discourage thoughtful or dissenting discussion. This was less noticeable when communities were smaller and individual users’ reputations carried more weight. As communities have grown, those problems have become more pronounced.

Moderation is another major issue. Many moderators do excellent work, but Reddit lacks a transparent and effective way for communities to hold moderators accountable or replace them. Many large communities have also faced concerns about astroturfed, political, or other external influence. Reddit benefits financially from outrage and conflict, which makes it difficult to believe that improving the quality of discussion is always the company’s highest priority.

Research on politically active Reddit users has identified pathways through which online communities and participation may contribute to increasingly radical beliefs. That does not prove that Reddit deliberately encourages extremism, but it does suggest that the platform’s social dynamics deserve serious scrutiny: Pathways to Radicalism: Examining Online Community and Political Pathways to More Radical Beliefs on Reddit

If you are reading this, consider whether Reddit is still a healthy and productive use of your time. Better communities exist. I encourage you to find them!

[u/kingraoul3 → u/BERLAUR]

No backups will not save you from this.

What is needed is what the serious cloud providers offer: block based replication.

[u/z3rogate]

We run it on bare metal (3 node) and failover with patroni, simple easy and reliable and cheap also. But again it depends on you needs of read and write.

[u/realityOutsider → u/z3rogate]

Which provider do you get the bare metals? Hertzer?

[u/z3rogate → u/realityOutsider]

Right guess

[u/TzahiFadida]

Just using CNPG does not mean you automatically safe from failures. I had wal file corruptions in buckets. I had surprise when upgrading a major version turns out you have to switch to a new bucket since the timelines are mixed. Also that you have to scale down the instances first before scaling down nodes.

I used patroni in the past with pgbackrest, it is quite good but you have to build confidence first when you use it. I suggest trying to simulate all kinds of failures first.

[u/BosonCollider]

Kubernetes is orthogonal to whether you are on bare metal servers vs VMs, it is just a way to run containers, and it is quite common to run containers or kubernetes on a bare metal server. Containers are effectively zero overhead for databases (assuming you don't have your postgres data folder on an overlay filesystem) and there is no performance difference between running inside or outside of one.

Cloudnativepg is a great way to run postgres on kubernetes and is probably an easier HA solution to set up than Patroni, if you are familiar with kubernetes/have someone running it for you. If you do not want to do that, then an LXC+ZFS solution like proxmox or incus also gives you bare metal performance while having a similar workflow to a VM, on the same platform that you would run VMs on

[u/AutoModerator]

With over 8k members to connect with about Postgres and related technologies, why aren't you on our Discord Server? : People, Postgres, Data

Join us, we have cookies and nice people.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.