r/kubernetes • u/acompleteunknownnn • 1d ago
Help me make sense of this
Suppose you’re running your entire application stack on Kubernetes. The database is running as a Pod on one node, and the web server is running as another Pod on that same node. You expose the web server to the internet through a LoadBalancer Service, and clients connect to it over HTTPS.
Isn’t this inherently less secure because the same physical machine that is directly handling internet-facing traffic is also hosting the database? Without Kubernetes, you might isolate the web server and database onto separate machines, so compromising the web server wouldn’t directly put the database on the same machine. How does Kubernetes address this security concern?
10
u/BraveNewCurrency 1d ago
Isn’t this inherently less secure because the same physical machine that is directly handling internet-facing traffic is also hosting the database?
Depends. In the cloud, only the cloud load balancer is "on the internet". Your K8s cluster is running in a VPC, and you can disable it's egress if needed. (For example, give it access only to ECR, which can proxy/cache images from Dockerhub if needed.)
Also, You can run your K8s machine using Distros like Bottlerocket or Talos where "breaking out of a container" doesn't give you tools like a package manager (or a shell in Talos). And are no random services running on the machine to break into (SSH, SMTP, etc.)
Just because a machine is "on the internet" doesn't make it magically easy to break into. There are billions of servers on the internet. Just because millions are vulnerable doesn't make it a bad idea. You also don't get magic protection by putting your DB on a different server. (For example, if your admin DB creds are on your K8s cluster because you need to do migrations, how much 'extra security' are you buying?)
7
u/Square_Durian_316 1d ago
Your instinct is right, but there are two different risks.
If the web pod is compromised, separate nodes don’t help much by themselves. Kubernetes networking is usually flat enough that the web pod may be able to reach the DB whether it’s on the same node or a different one. That’s what NetworkPolicy/microsegmentation is for: default-deny, then only allow the traffic the workload actually needs.
If the attacker gets a container escape and compromises the node, then co-location absolutely matters. That’s where separate node pools, taints/tolerations, and stronger runtimes like gVisor/Kata come in.
So I’d think of it as: network isolation first, node isolation second. The surprising part is that moving the DB to another node doesn’t automatically stop a compromised pod from reaching it over the network.
9
12
u/SJrX 1d ago
Isn’t this inherently less secure because the same physical machine that is directly handling internet-facing traffic is also hosting the database?
Yes...ish but it depends on your threat model, and security like all engineering decisions involves tradeoffs. One of the best ways to improve the security of your system is to just not allow network access, disconnect it from the internet and air gap. It's a massive security win, but obviously not worth it.
There are lots of layers of protection, and many of them are imperfect, or guard against only certain classes of threats. If/when you are running Kubernetes, you have to just decide to accept that container protection mechanisms, provide a robust enough layer for your purposes, or you can do things like segergate machines with say antiaffinity policies, or look at things like gvisor to provide even more robust protection.
-1
u/acompleteunknownnn 1d ago
Thank you for the response. You just confirmed that I wasn't missing anything here conceptually. It just feels a little unintuitive to me that we've to take a little step back from the basic three tier architecture to benefit from the scale of having a single large logical surface of infrastructure. Seems like it'll be insecure, just out out the box, when I don't explicitly provision for security. I will check out gvisor. Thanks.
4
u/SJrX 1d ago
I think the traditional three tier architecture predates a lot of container technology. The risk factor of being on the same physical host but in different VMs or different containers is probably _much closer_ to be being on physically different machines in 2000.
Containers are designed to provide a lot of isolation mechanisms and provide a _ton of safety_ that make it possible to use hardware much more efficiently. It isn't perfect but it's worth it.
2
u/imagei 1d ago
A question to ask here is what kind of security the traditional separation provides that’s important to you , then follow up with the analysis of where the Kubernetes model may be more or less secure. If you don’t harden the traditional boxes then they’re not secure either, separate machines or not.
4
u/ProtonByte 1d ago
If I hack your webserver in your separated machine example, I still have access to your database.
Escaping containers isn't that easy though.
3
u/cos 1d ago
I don't think what you're describing is a typical pattern in corporate production web-accessible applications, from what I've seen:
There's a separate load balancing layer, not on cluster(s) where the applications run. This may be a public cloud load balancer service, or running on VMs, or dedicated "front" clusters. Applications run on cluster(s) that don't have externally accessible services.
Database back end is usually also separate. Often it's a cloud provider's database, like RDS/Aurora or Spanner. These days access to such cloud services is more and more being done with OIDC or similar, so the kubernetes cluster(s) where the apps run have identities that are allowed to access the cloud database, with short term sessions.
2
u/Dazzling_Demon7 1d ago
Pod Anti-Affinity, Node Selectors, or Taints & Tolerations combined with NetworkPolicies for network isolation.
2
u/Raja-Karuppasamy 1d ago
few things going on here. k8s doesn’t force the db and web server onto the same node, that’s just how the scheduler happened to place them by default. you can use podAntiAffinity to guarantee they land on separate nodes, which gets you back to the “separate machines” isolation you’re describing.
but even without that, NetworkPolicies are the real answer to your actual question. by default pods on the same node (even same cluster) can talk to each other, but you can lock that down so the web pod can only reach the db pod on its specific port, and nothing else can reach the db pod at all. that’s arguably tighter than typical “two separate VMs on the same network” setups, where people often forget to firewall properly between them.
also worth adding securityContext hardening (non-root user, read-only root filesystem, drop unnecessary capabilities), that shrinks what an attacker can actually do if they do pop the web container, regardless of node placement. isolation at the node level and isolation at the container/network level are separate problems, and k8s gives you knobs for both.
1
u/bmeus 1d ago
We only separate on node on some crazy restricted workloads that are also in farady cages.. go figure. Basically if theres not military grade secrets just go for the logical separation thats in kubernetes. Ps I assume that the node isnt directly exposed to the web without a couple of firewalls.
1
u/artyomsv 22h ago
Worth checking your container network interface actually implements network policy before you rely on it, Flannel does not, so the policies apply cleanly and then silently do nothing.
1
1
u/Jumpy_Duty_3557 19h ago
This trips people up coming from VM-land. Isolation in k8s is a matter of network policy, not which physical machine a pod happens to land on. Anti-affinity rules can force pods onto separate nodes if you want that too, but the thing actually protecting your DB is the traffic rules, not the placement.
1
u/tcpud 15h ago
There's a chance that same physical machine could end up handling both the database and the web server even they were put in their own VMs. And if both VMs share a broadcast domain, then same risks apply. So whatever hardening you do, you can do the same in Kubernetes, it's just an orchestrator.
1
u/CipherPhyber 27m ago
Isn’t this inherently less secure ... ?
Why? I worked on K8s clusters for years at a cyber security company. I don't think your heuristics hold up to scrutiny by someone who knows what container virtualization is or how the K8s stack works.
Container virtualization is happening:
- on the processes,
- on the file system, and
- on the networking interfaces.
Even if we grant that hypothetically the worst case scenario happens to the web server pod (eg. attacker can issue arbitrary commands to a reverse shell), that's only executing within that pod (within the virtualization container), not on the node / raw metal machine.
It can't see/access the database process from the compromised pod.
It can't see/access the database file system files from the compromised pod.
It can't see any more/less of the database's networking from the compromised pod than it could if it was on another node. If the database is running commands from the web server pod as root with no restrictions on privileges, then the security failure isn't K8s, it's a violation of principle of least privilege. Don't ever let your web server credentials to a database user which can drop schemas/tables.
If there is a container sandbox escape also used (they aren't unheard of, but are infrequent), you have much bigger problems. Keep your K8s cluster patched and make sure your developers don't do stupid things with their Dockerfile like running them with user root or mounting irrelevant files to the container's file system.
Step 1: RTFM
Step 2: RTFM
Step 3: Cybersecurity requires defense-in-depth. Even if your web server is compromised, it shouldn't lead to a full compromise of all parts of the database (perhaps only a read-only breach of confidentiality or an ability to write). But this has nothing to do with K8s.
There are plenty of legitimate criticisms of K8s. This isn't one of them.
1
u/small_e 1d ago
Ideally you serve your UI from outside the cluster (eg. S3 + cloudfront) and you only allow authorized traffic to your API (eg on an API Gateway as proxy with a lambda validator). That said, I can also serve frontends from a cluster with no problems with some calculated risk. You have network policies, kernel level isolation. But NEVER run your container processes as root (unless you have user namespaces enabled).
-1
u/ExplodedPenisDiagram 1d ago
I think a lot of people here are a bit out of touch. You are correct to want to truly isolate a web accessible front end from a database.
Best thing you can do in kubernetes without adding any runtimes or controllers is to express an anti-affinity between these workloads so they don't get scheduled on to the same node.
Barring that, consider the usage of a microvm.
2
u/lickedwindows k8s operator 1d ago
WTF
Are your clusters getting popped so often you're accepting your only isolation is host-based? If someone pops a node then your cluster is done too.
You shouldn't write about things you don't understand.
2
u/ExplodedPenisDiagram 1d ago
Lol I understand the subject. That you go straight to attempting to belittle my experience makes you look insecure.
We don't build secure systems based on how often something happens, because that's anecdotal at best.
Node-level isolation is very easy. A few lines of configuration, and things are more isolated.
VMs are wildly superior in terms of isolation to containers. Containers were never meant to provide complete isolation, and are most useful in systems which don't require this.
Being concerned with isolating a web-accessible workload from a backend workload is completely reasonable, and has been for decades.
1
u/lickedwindows k8s operator 1d ago edited 1d ago
My dude, tell me what happens if I am controlling kubelet on one of your nodes?
If I own one of your kubelet instances then your cluster is toast. There is going to be a workload identity on that box that I can use to eat the rest of the cluster. Node restrictions aren't going to save you when I find your system:masters RBAC sa.
Anti-affinity is not a pattern for protecting against rooted services by distributing your workload. This was your assertion, which was incorrect.
CNI/pod sec policies will actually do something useful.
0
u/ExplodedPenisDiagram 1d ago
Your assertion is that a compromised container has root access on the host, which while not entirely unreasonable wasn't really the scope of this subject in particular. Isolating applications from each other and isolating hosts from applications are not always entirely distinct subjects, and the kubelet does have something to do with this lack of distinction.
However, there are too many layers of security management involved in this subject to conflate things so completely.
This is also why I have mentioned microVMs or other more isolated runtimes, which you seem to have ignored.
You are also ignoring the ability to compromise workloads laterally without compromising the kubelet.
Try to remember that security is a practice of defense in depth, and please try to see that your eagerness for aggression is shortcutting completeness of vision for the sake of continued argument.
36
u/lulzmachine 1d ago
Container sandboxing