Stop Relying on Default EKS Node Termination Handlers

The "Silent" Deployment Failure

Last month, we had a service intermittently dropping requests for exactly 30 seconds every few hours. The logs showed no pod crashes, no memory pressure, and no spikes in latency from the database. It looked like a network hiccup that we couldn't pin down.

I checked the CloudWatch metrics for the EKS node group and noticed a pattern. The latency spikes aligned perfectly with the termination of Spot instances. We were running our production workers on Spot to save 70% on compute, but we hadn't properly configured how the cluster handled the two-minute warning AWS sends before reclaiming an instance.

The Kubernetes scheduler was trying to route traffic to pods on a node that was already effectively dead. The service was essentially sending traffic into a black hole until the readiness probes finally caught up and removed the endpoints.

How AWS Spot Interruptions Work

When AWS decides to reclaim a Spot instance, it sends a notification to the instance metadata service. You have exactly 120 seconds to drain your pods, finish inflight requests, and shut down gracefully before the instance is forcefully terminated.

By default, a standard Kubernetes node doesn't know this is happening. The Kubelet will eventually realize the node is unreachable, but that process takes minutes—far longer than the two-minute window AWS gives you. If you aren't explicitly listening for that signal, you are going to drop every request currently in flight on that node.

Why You Need the AWS Node Termination Handler

You shouldn't write custom scripts to poll the metadata service. Instead, you need an automated handler. The AWS Node Termination Handler runs as a DaemonSet on your cluster. It watches the EC2 Instance Metadata Service (IMDS) for the termination notice and, upon receiving it, immediately cordons and drains the node.

The Configuration Trap

Most teams install the handler and assume it works. But if your terminationGracePeriodSeconds in your pod spec is longer than the time it takes to drain, or if your application doesn't handle SIGTERM signals correctly, the handler can only do so much.

If you don't have a preStop hook, your application process might exit immediately when it receives SIGTERM, cutting off connections before the load balancer has time to realize the pod is gone.

Add this to your deployment manifest:

spec:
  containers:
    - name: app
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 10"]
  terminationGracePeriodSeconds: 30

The sleep 10 isn't a hack; it's a buffer. It gives the Kubernetes control plane enough time to propagate the "NotReady" status to the Service and Ingress controllers while your application finishes processing the requests it already accepted.

What I Actually Do / My Take

I treat Spot instances as "ephemeral by default." If a service can't handle a sudden node loss, it doesn't belong on Spot.

1. Mandatory Termination Handler: Every EKS cluster I manage has the aws-node-termination-handler installed via Helm. I don't care if the cluster is small; it’s a non-negotiable insurance policy.
2. Pod Disruption Budgets (PDBs): I always define a PDB. If I have 3 replicas, I set minAvailable: 2. This prevents the termination handler from killing pods too aggressively if the cluster is already under pressure.
3. Graceful Shutdown Logic: I verify that every backend service actually catches SIGTERM. If your Node.js or Go server just kills the process immediately, you aren't doing "graceful" anything. You must stop accepting new connections, finish existing requests, and then exit.
4. Monitoring: I alert on node_termination_handler_interruptions_total. If the interruption rate is consistently high, it’s a sign that our chosen instance types are too constrained. I switch to "capacity-optimized" allocation strategies instead of picking specific instance families.

Don't settle for "it usually works." In production, "usually" is just a waiting room for an incident. If you're going to save money with Spot instances, spend a fraction of that time ensuring your infrastructure handles the volatility automatically.

Closing / TL;DR

Spot instances are great for cost, but they are a liability without automation. Install the aws-node-termination-handler, implement a preStop hook to delay pod exit, and set a PodDisruptionBudget. Your load balancer needs time to drain connections—don't let your code exit before the network catches up.

Tags: kubernetes · aws · eks · devops · spot-instances · cloud-cost-optimization · infrastructure-reliability · scc-architecture · cloud-native · production-readiness