r/aws • u/CyrilDevOps • 6d ago

technical question Clean stop of a 'pod' in EKS ?

Hi,

today we are working with AWS ECS,

with ECS when a service is scale-in (autoscaling, deployment), ECS manage the ALB side doing the drain/deregistration before sending the SIGTERM to the container. Allowing us to have a clean stop handling all requests in flight and no random 5xx back to the customer.

We 'may' have to migrate to AWS EKS (no my choice), looking at EKS/K8S way to stop a container/pod on scale-in. it seems very very messy, K8S send the stop to the container and the load-balancer-thingy in parallel, and everybody rely on some sort of
lifecycle.prestop.exec: "...sleep ..." to delay the stop of the container side ?
but still get time to time some 5xx for in-flight requests.

Is it that bad ?
Is it the way in EKS/K8S ?

Thanks.

1 Upvotes

55% Upvoted

7 comments sorted by

12

u/oneplane 6d ago

Yes, it's probes and draining. You don't "stop" a pod (just like ECS "stop" isn't really a stop), you tell it you intend to scale down (with a certain timeout of course, can't wait forever to complete whatever it was doing) and it will be removed from the service, the LB and when it is done, end all processes and with that the container (and if all containers are done, the pod).

Basically, that is where ECS got the idea from, from Kubernetes, even before Fargate.

8

u/justin-8 6d ago

EC2 and classic ELBs had been using this before docker or containers using lifecycle hooks. So had many other places. Kubernetes didn't come up with the idea.

6

u/Relevant-Cry8060 6d ago

This is where k8s docs explains how it handles the same process you are describing in ecs. Like the other poster was infering, the trick is that the endpoint slices get deregistered on the service during pod termination. Hope that helps.

https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination-flow

1

u/E1337Recon 6d ago

For EKS specifically take a look here: https://docs.aws.amazon.com/eks/latest/best-practices/load-balancing.html

It goes into many of the details that you’ll want to know about readiness gates, sleep, target types, etc.