A team moves its API to Kubernetes and switches to rolling updates. Deployments are smoother than before, but every release now comes with a small, predictable blip: for about a minute, roughly 1% of requests fail with 502 or 503 errors. Nobody is quite sure why, since Kubernetes replaces pods one at a time and the new ones pass their health checks. The explanation lies in two moments that rolling updates do not handle for you: when a new pod starts receiving traffic, and when an old pod stops.
What a rolling update actually does
A Deployment’s rolling update creates new pods and removes old ones in steps, controlled by two settings. maxSurge is how many extra pods may exist above the desired count during the rollout, and maxUnavailable is how many may be missing. Both default to 25%. With 8 replicas, Kubernetes may run up to 10 pods during the update and allow as few as 6 to be ready.
The controller only knows a pod is “ready” because a readiness probe says so, and it only knows a pod has “stopped” when the container exits. Everything between those signals is the application’s responsibility. Most deploy-time errors come from gaps at one of the two ends.
The starting end: ready means ready
Without a readiness probe, Kubernetes considers a pod ready as soon as its containers start. A Spring Boot or Django application may take 20 seconds after that to open its port, connect to the database and warm up. Requests routed to it in that window fail.
A readiness probe fixes this, but only if it checks the right thing. A probe on a static endpoint that returns success the moment the web server binds can still pass before the application can serve real requests. A good readiness endpoint confirms that the app has initialised its critical dependencies, such as a database pool. Be careful not to overdo it: if readiness fails whenever a shared dependency hiccups, every pod can become unready together, turning a partial problem into a full outage.
Keep liveness probes separate and simpler. Liveness answers “should this container be restarted?”, and making it depend on the database means a database blip restarts your whole fleet.
The stopping end: a race nobody tells you about
When Kubernetes decides to terminate a pod, two things happen roughly in parallel. The pod is marked as terminating, which causes it to be removed from the Service’s endpoints, and the kubelet runs any preStop hook and then sends SIGTERM to the container.
The word that matters is “parallel”. Removing an endpoint is not instantaneous everywhere. The change has to propagate to kube-proxy rules on every node, to ingress controllers, to service meshes and, if you use a cloud load balancer that targets pods directly, to the cloud provider’s API. That can take a few seconds. Meanwhile, if the application exits immediately on SIGTERM, there is a window in which traffic is still being sent to a pod that has stopped listening. Those requests produce 502s and connection resets.
The standard fix looks almost silly: delay the shutdown. A preStop hook that sleeps for 5 to 15 seconds keeps the application running and serving while the rest of the cluster catches up and stops sending it traffic. Recent Kubernetes versions also offer a built-in sleep action for preStop hooks, so you do not need a shell and a sleep binary in the image. After the delay, SIGTERM arrives and the application can shut down for real.
What the application must do on SIGTERM
Graceful shutdown has a definite sequence:
- Stop accepting new connections, or start failing readiness so no new traffic is routed.
- Let in-flight requests finish, up to a deadline.
- Close idle keep-alive connections cleanly, for example by responding with a connection-close header, so clients do not send a request down a socket that is about to disappear.
- Stop background consumers so they do not pick up new queue messages, and acknowledge or release the ones they hold.
- Flush logs and metrics, close database pools, and exit.
Most frameworks support this but some need it switched on. Spring Boot, for example, has a graceful shutdown setting with a configurable timeout per shutdown phase. Node.js servers need explicit handling of the signal and a call to close the server.
The whole sequence, preStop delay plus drain time, must fit inside terminationGracePeriodSeconds, which defaults to 30 seconds. When that runs out, the kubelet sends SIGKILL and the process dies mid-request. If you have long-running requests such as file uploads or report generation, raise the grace period or move that work to asynchronous jobs.
Small details that break it anyway
- The signal never reaches the app. If a container’s entrypoint is written in shell form, a shell runs as process 1 and may not forward SIGTERM to your application. The app is then killed abruptly at the end of the grace period. Use the exec form, or a tiny init process such as tini.
- The load balancer has its own timer. Cloud load balancers have deregistration or draining delays (AWS target groups default to 300 seconds) that interact with your grace period. If the pod exits before the load balancer stops sending traffic, you get errors again.
- Not enough spare capacity. With
maxUnavailableat 25% and pods already running hot, losing a quarter of capacity mid-rollout can overload the rest. Setting it to zero with a positive surge keeps full capacity throughout, at the cost of temporarily needing more resources. - Incompatible versions side by side. During a rollout, old and new versions serve traffic together. If the new version changes the database schema or an API response in a breaking way, the old pods fail. Schema changes need to be backward compatible across at least one release.
- Client-side connection reuse. Callers with long-lived pooled connections may keep reusing a connection to a terminating pod. Short idle timeouts on client pools and retries for idempotent requests on connection errors cover the remaining gap.
Proving it works
Zero-downtime deploys should be tested, not assumed. Run a steady load test against a staging environment, perhaps a few hundred requests per second, and trigger several deployments during it. Count every non-2xx response and every connection error. Once that count is reliably zero across repeated rollouts, add the same check to your release pipeline so a later change, such as a new base image with a different entrypoint, cannot quietly bring the errors back.
For changes where even careful rolling updates are too risky, blue-green deployments (switching all traffic between two complete environments) and canary releases (sending a small percentage to the new version first) add further control. But both still depend on the same foundation: instances that only receive traffic when ready, and that leave without dropping what they were doing.
Written by the RP Strativa Team. Replica counts and traffic levels in the examples are illustrative.
Leave a Reply