Stop Ignoring Kubernetes Liveness and Readiness Probes

The "It Just Reboots" Mystery

I spent an entire morning once chasing a "ghost" memory leak in a Node.js service. The logs showed the process exiting with SIGKILL, and Kubernetes dutifully restarted it. I spent hours analyzing heap dumps and inspecting garbage collection stats.

The actual culprit? A poorly configured liveness probe that was too sensitive for the application’s startup time.

The application was taking 45 seconds to initialize its connection pool to Postgres. The liveness probe was configured with a 5-second timeout and a 10-second interval. Kubernetes thought the app was dead, killed it, and started the cycle over. My service spent more time restarting than serving traffic.

Readiness vs. Liveness: Why You Need Both

Too many teams treat these probes as synonymous. They aren't. They serve completely different stages of the application lifecycle.

Liveness Probes: "Is the process stuck?"

A liveness probe tells the Kubelet if your application is deadlocked or in a state where it cannot recover without a restart. If this fails, Kubernetes kills your container.

• Use case: Deadlocks, infinite loops, or crashes that don't trigger an automatic exit.
• The Trap: Do not put heavy dependency checks here. If your database is momentarily slow, and your liveness probe checks that database, you will trigger a cascading restart of your entire cluster.

Readiness Probes: "Are you ready to handle traffic?"

A readiness probe tells the Kubelet when your pod is capable of accepting traffic. If this fails, the endpoint controller removes the pod from the Service and Ingress load balancer rotations.

• Use case: Warming up caches, establishing database connections, or running migrations.
• The Trap: If you don't define this, Kubernetes starts sending traffic to your container the millisecond the process starts, resulting in 502 Bad Gateway errors while the app is still booting.

Why Default Settings Are Dangerous

Kubernetes defaults are generic; they don't know your app. Here is a configuration that has caused production outages for teams I’ve mentored:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 0
  periodSeconds: 10

This is a liability. Without an initialDelaySeconds or startupProbe, you are racing against the Kubelet. If your app takes 15 seconds to bind to the port, the probe will fail twice and kill your container before it ever hits its first request.

The "Startup Probe" Pattern

If you have a slow-starting application (like a heavy JVM app or a Node.js service with massive dependencies), use a startupProbe.

A startup probe disables liveness and readiness checks until the pod has finished its "boot" phase. Once it passes, the regular liveness and readiness probes take over.

startupProbe:
  httpGet:
    path: /health/ready
    port: 8080
  failureThreshold: 30
  periodSeconds: 10
# This gives your app 300 seconds (5 minutes) to finish starting up
# without being killed by the liveness probe.

What I Actually Do

I follow a strict hierarchy for production workloads:

1. Liveness is for emergencies only: My liveness endpoint is usually just a simple return 200. It only checks if the process is alive. If the database is down, I do not want the pod to restart. Restarting a pod won't fix a database connection issue; it just adds unnecessary load to the API server and the database itself.
2. Readiness is for logic: My readiness endpoint checks the critical path—database connectivity, cache availability, and external API dependencies. If these are failing, the pod should be taken out of rotation.
3. Use initialDelaySeconds conservatively: I prefer startupProbe over guessing the timing. It’s cleaner and less prone to configuration drift as the app grows.
4. Monitor the failures: I alert on KubePodNotReady and LivenessProbeFailure events. If a pod is failing its liveness check, I want to see the logs before the pod is nuked and the evidence is gone.

TL;DR

Don't let Kubernetes manage your app’s health with defaults. If your pods are flapping, you probably have a liveness probe that is too aggressive. Use a startupProbe for initialization, a simple "is-alive" check for liveness, and a robust dependency check for readiness.

If your application crashes when the database is slow, you’ve built a self-destruct mechanism, not a resilient system.

Tags: kubernetes · devops · infrastructure · readiness-probes · liveness-probes · cloud-native · site-reliability · production-engineering · eks · container-orchestration