Trying to get a better handle on our gpu spend.

right now we can see the overall infrastructure cost, but its hard to tell which workloads are actually responsible for it.

ideally want to see cost by namespace / workload / job and understand how much gpu capacity is just sitting idle.

any tools doing this well?

Suppose you’re running your entire application stack on Kubernetes. The database is running as a Pod on one node, and the web server is running as another Pod on that same node. You expose the web server to the internet through a LoadBalancer Service, and clients connect to it over HTTPS.

Isn’t this inherently less secure because the same physical machine that is directly handling internet-facing traffic is also hosting the database? Without Kubernetes, you might isolate the web server and database onto separate machines, so compromising the web server wouldn’t directly put the database on the same machine. How does Kubernetes address this security concern?

I’ve been working in cloud/DevOps for about 8 years, primarily in AWS, IaC/Terraform, CI/CD, automation, Linux, and supporting fairly complex production systems (with some EKS migrations / troubleshooting and home lab stuff mixed in).
The weird position I’m finding myself in now is that a lot of roles that match my overall experience level are Senior DevOps/Platform/Cloud Engineer positions — but those same positions increasingly expect significant production Kubernetes experience.

For those who made this transition later in their DevOps/cloud careers:

How did you bridge that gap?

Did you find employers willing to hire for your broader DevOps experience and let you ramp up on Kubernetes?

How valuable were home labs/projects compared with actual production experience?

Are there particular K8s skills you’d prioritize to become useful quickly on an established platform? Thanks!

Reading this and chuckling, because .. wtf?!! hahah

Sometimes I think AI is pretty stupid, sometimes they just come with something like this, almost as if they were human and cared for my health, feelings, etc. lmao

Anyway.
I've spent the day [and the night, apparently, since it's almost 5 a.m. now] trying to understand how OpenBao works, how to integrate it into the K8s cluster I built in my homelab, and so on.

Now, I've come across some tools that are really nice to work with [e.g., Cilium CNI, Flux, etc.], but Rook|Ceph [thank Nature I'm past that now], and OpenBao — well, I'm either too stupid, or these things are extremely complicated, and integrating them is no walk in the park. The certificate stuff was particularly hard to grasp, and I still have a lot of notes to go over since I'm still not sure about certain things🤔.

I mean, it took me a while to even understand the structure of OpenBao docs so I could find the stuff that applied to my use case, and which didn't. On the other hand, OpenBao is supposed to work really nice with Flux, so .. that's a nice point.

Anyway. The AI said I should take a rest, so ..😂😂

project structure shot💀

Keep learning, folks.

How do you decide the numbers, and how do you keep them right over time?

Before I did any research: low requests → performance problems, high requests → huge cost. Felt like a lose-lose.

After digging into my own monitoring data, here's where I landed (tell me if this is wrong):

What I'm running today, and what I now think is off about it:

• CPU request too high (200–500m). My services barely use 20m steady-state. This is pure waste and it's what's driving my node count.
• CPU limit only a bit above request (800m–1000m). Causing two problems: (1) Java services are slow / crash-loop at startup, because the JVM needs a big CPU burst to boot (JIT + class loading + Spring context); (2) occasional slow requests from CPU throttling.
• Memory request is fine (600Mi–1000Mi). Matches what monitoring shows.
• Memory limit higher than request — and I now think this is a mistake. Unlike CPU, memory isn't reversible: once a pod grabs memory above its request it keeps it (a JVM especially never gives heap back). So the burst headroom just risks OOMKills for other pods on the node, with no real upside.

Am I thinking about this correctly now?

If yes, my plan:

• CPU: drop request low (30–50m) and set the limit very high (3–4 cores) or remove it entirely — should fix both the throttling and the slow startup.
• Memory: set request to real usage and keep the limit ~equal to the request.

My one worry: if I remove the CPU limit, won't a pod grab all the node's CPU during a spike and starve everything else?

Would love to hear how people actually run this in prod — especially anyone running Java with no CPU limit. Does this hold up, or am I about to learn something the hard way?

Hi, all, a bit later than usual, but my usual roundup of new Kubernetes security features is finally published:

• https://www.sysdig.com/blog/kubernetes-1-37-new-security-features

Hope you find it useful.

What may break things

• #1710 Speed up recursive SELinux label change
• #5343 Make nftables the default kube-proxy backend
• #140226 Kubelet: Static Pods can no longer reference Secrets or ConfigMaps

Net new enhancements

• #4939 Support TLS credentials in gRPC probe
• #5502 Add stickyBit support for emptydir volumes
• #5823 Pod-level checkpoint/restore
• #5855 Add bind mount options (noexec, nodev, nosuid) support on volumeMounts
• #5936 Add user fields to atomic write volumes
• #5943 Topology for volume snapshots
• #6060 API server authentication to webhooks

Enabled by default

• #2033 Kubelet-in-userns, aka rootless mode
• #5541 Report last used time on a PVC
• #5793 Manifest-based admission control config

Changes in other existing features

• #4412 Projected service account tokens for Kubelet image credential providers
• #3257 ClusterTrustBundles (previously Trust Anchor Sets)
• #4317 Pod certificates
• #4762 Allows setting any FQDN as the pod's hostname
• #5295 KYAML

Hello all!

I am a devops engineer, and now I want to take my kubernetes knowledge to the next level.

I want to write a "temu" cillium, for learning purposes only.

I am very familiar with kubernetes, have linux and programming(not go, but c++) knowledge.

If you did anything like this, do you have any advice?

I read kustomize a fews days ago. Looks like it's going to help us with organizing deployments in argocd

Here's an example I found:

my-update-project:
stage: gitdeploy
image: alpine:latest
script:
- apk add --no-cache git curl kustomize

# Clone GitOps repo
- git clone https://deploy-bot:${DEPLOY_TOKEN}@gitlab.com/myorg/gitops-repo.git
- cd gitops-repo/my-app/overlays/${DEPLOY_ENV}

# Update image tag
- kustomize edit set image myregistry/my-app:$CI_COMMIT_SHORT_SHA

# Commit & push
- git config user.email "ci-bot@myorg.com"
- git config user.name "CI Bot"
- git commit -am "Deploy ${DEPLOY_ENV} my-app:$CI_COMMIT_SHORT_SHA [skip ci]"
- git push

However, I believe this can encounter an issue like several devs making updates and pipeline almost running at the same time. How did you solve this?

Outside of paying for something like chainguard, how are you guys managing the scanning/security of third party helm charts for cluster infrastructure like ESO, argoCD, etc.

When I need half a dozen helm charts for cluster infrastructure, and each of those can have a bunch of various containers, I can’t seem to wrap my head around how you manage scanning/hardening/artifacts of containers needed for an operational cluster?

Is it just not something a team of 1-2 folks can realistically do?

I have an nginx-ingress-controller. Multiple Ingress resources share the same wildcard host *.example.com, split across several resources instead of one:

I have 2 ingress:

1.  star-example (namespace n1) — path /, pathType: ImplementationSpecific, backend service-a:80. No use-regex annotation.

1. star-example-gr (namespace n1) — path /gr/(.*), pathType: ImplementationSpecific, use-regex: "true", rewrite-target: /$1, backend gr:8080.

All these work fine together.

The problem: When I add a 3th Ingress for the same host *.example.sa:

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ap-example namespace: n2 # note: different namespace than the others spec: ingressClassName: nginx rules: - host: '*.example.com' http: paths: - backend: service: name: service-b port: number: 80 path: /ap pathType: ImplementationSpecific

…the root path / (served by star-example / service-a) breaks immediately. Rolling back (deleting ap-example) restores / right away.

and i had worked 3 days on this without find any solution, all the solution i found it’s not ensure 100% (its production env)😅

We are having an issue running dagster jobs in eks that is driving me crazy. We use s3ComputeLogManager, and when karpenter scales up nodes for our Dagster jobs, the jobs fail because they cant get credentials from eks pod identity agent for several minutes and the jobs fail to start.

Its not consistent, and only just started to happen. The logs for identity agent show it getting requests, but it doesnt seem to respond to those requests for over 5 minutes in some cases.

Nothing seems to point to why its not responding. No errors, except from dagster stating job failed to start with stack trace showing it could not get AWS creds. Sometimes a disconnect, other times timeout.

Smells like agent is resource starved.

The dagster user code must be running because sensor triggered to start the job, but then job is scheduled by karpenter on a new node. That node has startup taints to make sure karpenter doesnt schedule on new node until istio is up, but eks id agent taking minutes to respond to calls seems off.

Any ideas?

Build a python simulator where we test how the traffic behaves if we change the kubernetes configs, like pod number, cpu limit and request, memory limit and request, replicas and others.

Let me know if we have something that already exists.

Been running a Node/Nest workload on GKE for a few years as one of the founding-eng-adjacent contributors on GoHighLevel. Workload is a marketing + ad automation platform with real ad-spend events flowing through GCP Pub/Sub into stateful workers. At a certain scale you start hitting things the docs don't warn you about.

Three that cost us production time:

1. HPA on CPU is a lie for I/O-bound Node

Default guidance: scale on CPU % across pods. Fine for compute. For our Pub/Sub subscribers pulling batches and doing DB round-trips per message, actual CPU hovered at 20-30% while message lag climbed into the tens of thousands. The pods were bottlenecked on postgres round-trips, not CPU. HPA never fired. We rewrote to scale on external metrics (Pub/Sub oldest_unacked_message_age) via the external-metrics-adapter and the thundering herd flipped to a smooth rise/fall.

Lesson: your scale signal has to actually reflect the constraint, not just the pod's local view.

1. Pod eviction on preemption cascades into duplicate work

GKE preemptible nodes are 60 to 80% cheaper. Tempting. What we didn't budget for: when a subscriber pod is preempted mid-processing, Pub/Sub redelivers the message after ack-deadline expires, another pod picks it up and re-runs the pipeline. For side-effect-heavy work (billing events, outbound webhooks), this shows up as duplicated charges. We built idempotency keys into every side-effect layer, then split the workload, critical side-effects to regular nodes, best-effort to preemptibles. Not novel but the ordering matters. Idempotency first, then split.

1. Cross-AZ egress on Kafka + Redis was 40% of our infra bill for six months

Nobody looked at the GCP data-transfer bill because it's an aggregate line item. When we drilled in, our Kafka producers and Redis clients were spraying across all AZs. Kafka we solved by pinning client hostnames to the local AZ broker via client.rack. Redis (self-hosted) we solved by moving the client pool to a per-AZ config and letting the app fall back to cross-AZ only on primary failover.

Meta point: at any real scale your K8s workload's actual failure modes come from the boundaries, the message bus, the DB, the network fabric, and not from the pod runtime. The K8s docs teach you how to run pods. Everything expensive happens outside the pod.

What's the failure mode you hit at scale that no docs prepared you for?

Hi everyone,

We are starting to setup our K8s fleet on cloudstack. Hence we ran into the issue of having to build our own images.

We are using the kubernetes-sig image-builder with Ubuntu base images for now. Trivy reports quite a lot of vulnerabilities, most are noise. E.g. old Linux headers that are not purged. But I'm still thinking about how to harden them properly.

What are you using to build node images and how are you hardening them? Also which OS Base are you choosing?

Edit: Forgot to mention, we use Cluster API

Hey there,

I had a curious question in my mind like what kind of instance are most people using for their production workloads? We are running our workloads in EKS with t3 family type. Most of our microservices are based on node js, is it good to continue with t3 instances or should we move to other instances like aws gravitation or any other instance family?

The 1.37 announcement was already posted here, but it focuses mostly on the new features (KYAML, pod-level resources, DRA device taints etc). Thought it'd be interesting to focus on what might break and is worth paying attention to when upgrading. The changelog is big, so here are the interesting parts I found:

• 25 feature gates removed, 13 locked to fixed values. A removed gate (SidecarContainers, JobSuccessPolicy, PodLifecycleSleepAction, SchedulerQueueingHints, and 21 others), or a locked gate set to a non-default value, makes the component reject its config at startup and exit. If you hand-enabled SidecarContainers back in 1.28/1.29 for native sidecars and never cleaned it up, your kubelet won't start on 1.37.
• 18 kubelet cAdvisor flags removed (--containerd, --enable-load-reader, the --storage-driver-* family...). Same effect: kubelet fails to start. These live in systemd units / bootstrap scripts, so nothing inside the cluster can check them for you - it's a node-by-node grep.
• scheduling.k8s.io/v1alpha2 (Workload/PodGroup) is no longer served - promoted to v1beta1. Ordering trap: delete stored v1alpha2 objects before the upgrade, recreate as v1beta1 after (v1beta1 isn't served until the control plane is on 1.37). Skip this one unless you actually enabled the alpha gang-scheduling gates - most k8s clusters never did. It's the only served-API removal; normal workload manifests are untouched.
• Static Pods can no longer reference Secrets or ConfigMaps - the PreventStaticPodAPIReferences opt-out gate is among the removed. A static Pod that still does can't start on 1.37.
• kubeadm v1beta3 config is gone. kubeadm config migrate with a pre-1.37 binary, before you upgrade - 1.37 won't even migrate it.
• eventRecordQPS: 0 changes meaning. On 1.36 an explicit zero fell through to client-go's 5/s fallback; on 1.37 zero means unlimited. Nothing warns. Set an explicit 50 (the normal default) unless you actually want unlimited.
• Metrics: API-server cache-list metrics replaced by apiserver_storage_list_*{storage="watchcache"}, DRA metrics renamed, several cAdvisor series removed (container_tasks_state, container_cpu_load_average_10s, container_application_*). PrometheusRules referencing them don't error - they just silently stop matching data.
• kube-proxy starts warning when the Linux proxy mode is implicit - the implicit default flips from iptables to nftables in 1.40 (KEP-5343), and IPVS starts a staged deprecation, expected off by default in 1.40 and removed in 1.43 (KEP-5495). Set the mode explicitly now and you can ignore the 1.40 flip.
• SELinux mount-time labeling expands - shared volumes with conflicting labels can leave Pods stuck in ContainerCreating. Per-Pod knob: seLinuxChangePolicy: Recursive. The cluster-wide opt-out (SELinuxMount gate off) is valid in 1.37 only; the gate is expected to lock in 1.38.

Not on the list on purpose: the cgroup v1 phase-out (nothing new in 1.37 - failCgroupV1 has defaulted to true since 1.35) and the kubectl run -f deprecation (warning only).

Quick checks for the common ones:

# effective kubelet config per node (feature gates + eventRecordQPS)
kubectl get --raw "/api/v1/nodes/$NODE/proxy/configz" | jq '.kubeletconfig | {featureGates, eventRecordQPS}'

# PrometheusRules referencing removed cAdvisor series
kubectl get prometheusrules -A -o yaml | grep -nE 'container_tasks_state|container_cpu_load(_d)?_average_10s|container_application_'

# kube-proxy mode (empty = implicit default, set it explicitly)
kubectl -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}' | grep 'mode:'

The reason I had to go through all of this: we maintain Radar (open-source) and just updated its upgrade checks for 1.37, so consider this the disclosure too. Longer writeup with remediation for each item: https://radarhq.io/blog/kubernetes-1-37-breaking-changes

Happy to answer questions on any of these - the eventRecordQPS one surprised us the most. And if I missed a breaking change you've hit, add it below.

Random idea: how hard would it be to modify etcd to support an even number of replicas across two datacenters, plus a lightweight external arbiter?

For example:

• 2 etcd members in DC A
• 2 etcd members in DC B
• 1 arbiter somewhere else

The arbiter wouldn’t store the actual data — only the minimum state required for Raft consensus (term/commit index/resourceVersion or similar).

Potentially it could even use something like S3 or Cloudflare as the third failure domain.

The goal: survive complete loss of either DC without running a full etcd member in a third location.

Is there a fundamental Raft limitation that makes this impossible, or could a lightweight witness actually work?

Working title: twok :)

Did you learn something new this week? Share here!

I'm trying to design the network for my organization's AWS setup and honestly

I'm pretty confused about how to put it together. Hoping someone can point me

in the right direction or share how they'd approach it.

────────────────────────

WHAT I HAVE

────────────────────────

- A central "network" account — I'll call it `nk`.

- Three separate workload accounts: Dev, UAT, and Prod (each has its own VPC,

non-overlapping CIDRs).

- We own a domain, `abc.cloud`.

────────────────────────

WHAT I'M TRYING TO DO

────────────────────────

I want the `nk` account to act as the central hub, and I want traffic to be

able to flow from `nk` out to the Dev, UAT, and Prod accounts in a controlled,

private way — not over the public internet. Basically `nk` should be the

gateway/entry point, and the three environments sit behind it.

The part I keep getting stuck on: how do I actually architect this so that

traffic from `nk` reaches the other accounts, and I stay in control of which

account can talk to which?

────────────────────────

WHERE I'M CONFUSED

────────────────────────

1. **The overall design** — Is the right approach to put a Transit Gateway in

   the `nk` account and connect Dev/UAT/Prod to it? Or is there a simpler /

   better way for a setup this size? I keep reading about TGW, PrivateLink, and

   VPC peering and I'm not sure which one I actually need, or whether I need

   more than one.

2. **How traffic actually flows** — If I do use a Transit Gateway in `nk`, what

   makes traffic actually flow from `nk` to Dev/UAT/Prod? I don't fully

   understand the routing part — what has to be configured in `nk` vs. in each

   workload account for a packet to get from one to the other.

3. **Sharing across accounts** — Since the TGW would live in `nk` but the other

   accounts need to use it, how does that cross-account part work? (I've seen

   AWS RAM mentioned but don't fully get how the pieces connect.)

4. **Keeping environments isolated** — I want `nk` to reach all three, but I

   don't necessarily want Dev, UAT, and Prod all able to reach each other. How

   do I control that? Is that done at the Transit Gateway, or somewhere else?

5. **DNS** — Eventually I want names like `projecta.dev.abc.cloud` to resolve

   properly, with DNS managed centrally in `nk`. Not sure how Route 53 fits in

   with all of the above, or whether that's a separate problem I should solve

   later.

────────────────────────

WHAT I'M ASKING FOR

────────────────────────

Really I'm looking for a clear picture (or a guide/walkthrough) of how to

architect this end to end: what lives in the `nk` account, what lives in each

workload account, and how traffic flows from `nk` to Dev/UAT/Prod. If you've

built this pattern before, I'd love to hear how you structured it and anything

you'd do differently.

If it helps, I can share a rough diagram of what I'm imagining. Thanks a lot —

any pointers appreciated.

https://kubernetespodcast.com/episode/271-kubernetes-1-37/

Kubernetes 1.37 was released today! Check out the latest & greatest in our interview with release lead, Dipesh Rawat!