Health Check #

A health check is a mechanism for detecting non-functional backend servers and automatically removing them from the load balancing rotation. Without health checks, Nginx keeps sending requests to a down server until clients receive errors — a bad experience for users and hard to debug.

Understanding the difference between passive and active health checks is key to designing a reliable system.

Passive vs Active: The Fundamental Difference #

flowchart TD
    subgraph PASSIVE["Passive Health Check\n(Nginx Open-Source)"]
        direction TB
        P1["Client → Nginx → Server A\n(normal request)"]
        P2["Server A fails to respond"]
        P3["Nginx records the failure\nfail counter +1"]
        P4{"fail counter\n≥ max_fails?"}
        P5["Server A stays in rotation\n(next user could be affected)"]
        P6["Server A marked DOWN\nfor fail_timeout seconds"]
        P1 --> P2 --> P3 --> P4
        P4 -- "Not yet" --> P5
        P4 -- "Yes" --> P6
    end

    subgraph ACTIVE["Active Health Check\n(Nginx Plus only)"]
        direction TB
        A1["Nginx proactively sends\nGET /health every N seconds"]
        A2{"Response\n200 OK?"}
        A3["Server stays in rotation\n(clients unaffected)"]
        A4["Server immediately marked DOWN\n(BEFORE any client is affected)"]
        A1 --> A2
        A2 -- "Yes" --> A3
        A2 -- "No" --> A4
    end

Aspect	Passive (Open-Source)	Active (Nginx Plus)
Failure detection	After a client request fails	Before any client is affected
Impact on clients	Some clients hit errors first	Zero — proactive detection
Configuration	`max_fails` + `fail_timeout`	`health_check` directive
Probe endpoint	None	`GET /health` or custom
Cost	Free (open-source)	Nginx Plus (paid)

Passive Health Check: Complete Configuration #

upstream app_servers {
    # Shared memory for health check data across worker processes
    zone app_upstream 64k;

    # max_fails: how many failures within the fail_timeout window before marking the server down
    # fail_timeout: the failure window duration AND how long the server is "excluded"
    server 10.0.0.1:3000 max_fails=3 fail_timeout=30s;
    server 10.0.0.2:3000 max_fails=3 fail_timeout=30s;
    server 10.0.0.3:3000 max_fails=3 fail_timeout=30s;

    # Backup server — appears if all the servers above are down
    server 10.0.0.4:8080 backup;

    keepalive 32;
}

server {
    listen 443 ssl;
    server_name example.com;

    location / {
        proxy_pass http://app_servers;

        # Conditions counted as "failures" for max_fails
        # error: connection refused/timeout
        # timeout: proxy_read_timeout exceeded
        # http_500, http_502, http_503, http_504: errors from the backend
        proxy_next_upstream error timeout http_500 http_502 http_503 http_504;

        # Maximum total time to try all servers
        proxy_next_upstream_timeout 10s;

        # Maximum number of retries to other servers
        proxy_next_upstream_tries 3;

        proxy_connect_timeout 5s;
        proxy_read_timeout   30s;
    }
}

Setting the Right Values #

For normal web applications:
  max_fails=3 fail_timeout=30s
  → Server is removed after 3 failures within 30 seconds
  → Tried again after 30 seconds
  → Moderate tolerance, avoids false positives from momentary spikes

For critical applications (fintech, healthcare):
  max_fails=2 fail_timeout=15s
  → More aggressive, faster detection
  → Make sure monitoring is active because false positives are more likely

For applications with occasional GC pauses or slowness (Java, Go with GC):
  max_fails=5 fail_timeout=60s
  → High tolerance — a momentary timeout from a GC pause doesn't
    immediately take the server out of rotation

Default if not specified: max_fails=1 fail_timeout=10s
  → TOO AGGRESSIVE for most cases
  → A single timeout takes the server out for 10 seconds

Health Endpoints: Best Practices on the Application Side #

Although Nginx open source doesn’t do active health checks, you still need a good /health endpoint in every backend — for external monitoring, other load balancers, or Kubernetes readiness probes:

Node.js #

// Health check that verifies all critical dependencies
app.get('/health', async (req, res) => {
    const checks = {};
    let statusCode = 200;

    // Check the database connection
    try {
        await db.raw('SELECT 1');
        checks.database = { status: 'ok' };
    } catch (err) {
        checks.database = { status: 'error', message: err.message };
        statusCode = 503;
    }

    // Check the Redis/cache connection
    try {
        await redis.ping();
        checks.cache = { status: 'ok' };
    } catch (err) {
        checks.cache = { status: 'error', message: err.message };
        statusCode = 503;
    }

    // Check the message queue (optional — depends on whether it's critical)
    try {
        // Only check the connection, no need to consume messages
        checks.queue = { status: 'ok' };
    } catch (err) {
        checks.queue = { status: 'degraded' };
        // Don't set statusCode to 503 if the queue isn't critical
    }

    res.status(statusCode).json({
        status: statusCode === 200 ? 'ok' : 'error',
        uptime: process.uptime(),
        timestamp: new Date().toISOString(),
        checks
    });
});

Go #

func healthHandler(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
    defer cancel()

    checks := map[string]string{}
    statusCode := http.StatusOK

    // Database check
    if err := db.PingContext(ctx); err != nil {
        checks["database"] = "error: " + err.Error()
        statusCode = http.StatusServiceUnavailable
    } else {
        checks["database"] = "ok"
    }

    // Redis check
    if err := rdb.Ping(ctx).Err(); err != nil {
        checks["cache"] = "error: " + err.Error()
        statusCode = http.StatusServiceUnavailable
    } else {
        checks["cache"] = "ok"
    }

    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(statusCode)
    json.NewEncoder(w).Encode(map[string]interface{}{
        "status": map[int]string{200: "ok", 503: "error"}[statusCode],
        "checks": checks,
    })
}

Nginx for the Health Endpoint #

Expose the health endpoint on a separate port so it doesn’t mix with production traffic:

# Port 8080: dedicated to internal health checks and monitoring
server {
    listen 8080;
    server_name _;

    # Restrict access to internal networks only
    allow 10.0.0.0/8;
    allow 127.0.0.1;
    deny all;

    # Proxy to the application's health endpoint
    location /health {
        proxy_pass http://localhost:3000/health;
        access_log off;   # Don't log health check requests to the normal access log
        proxy_read_timeout 5s;
    }

    # Nginx's own status
    location /nginx_status {
        stub_status;
    }
}

Simulating Active Health Checks with OpenResty #

If using OpenResty (Nginx + LuaJIT), you can add active health checks with the lua-resty-upstream-healthcheck module:

# Install OpenResty and the module
opm get openresty/lua-resty-upstream-healthcheck

# nginx.conf (OpenResty)
http {
    # Shared memory for health check status
    lua_shared_dict healthcheck 1m;

    # Run the health checker in the background for each worker
    init_worker_by_lua_block {
        local hc = require "resty.upstream.healthcheck"

        local ok, err = hc.spawn_checker {
            shm = "healthcheck",    -- shared memory name
            upstream = "app_servers",   -- upstream block name
            type = "http",
            http_req = "GET /health HTTP/1.0\r\nHost: localhost\r\n\r\n",
            interval = 2000,        -- check every 2000ms
            timeout = 1000,         -- 1000ms timeout
            fall = 3,               -- 3 failures → mark DOWN
            rise = 2,               -- 2 successes → mark UP again
            valid_statuses = {200, 204},  -- status codes considered healthy
            concurrency = 10,       -- check all servers concurrently
        }

        if not ok then
            ngx.log(ngx.ERR, "Failed to spawn health checker: ", err)
        end
    }

    upstream app_servers {
        server 10.0.0.1:3000;
        server 10.0.0.2:3000;
        server 10.0.0.3:3000;

        keepalive 32;
    }

    server {
        # Endpoint to view the current health check status
        location /upstream_status {
            allow 127.0.0.1;
            deny all;

            content_by_lua_block {
                local hc = require "resty.upstream.healthcheck"
                ngx.say(hc.status_page())
            }
        }
    }
}

# View the health check status
curl http://localhost/upstream_status
# Output:
# Upstream app_servers
# Primary Peers
# 10.0.0.1:3000 UP
# 10.0.0.2:3000 DOWN    # already removed from rotation
# 10.0.0.3:3000 UP

Monitoring Upstreams with Prometheus #

For more comprehensive monitoring, use nginx-module-vts or the nginx-prometheus-exporter:

# nginx-prometheus-exporter: expose stub_status as Prometheus metrics
docker run -d \
    --name nginx-exporter \
    -p 9113:9113 \
    nginx/nginx-prometheus-exporter:latest \
    -nginx.scrape-uri=http://localhost:8080/nginx_status

# Scrape configuration in Prometheus
# prometheus.yml
scrape_configs:
  - job_name: nginx
    static_configs:
      - targets: ['localhost:9113']

Available metrics:

nginx_connections_active — number of active connections
nginx_connections_waiting — idle connections (keepalive)
nginx_http_requests_total — total requests

For per-upstream metrics, use nginx-module-vts, which provides:

Request count per backend server
Response time per backend server
Error count per backend server

Graceful Failover: Complete Production Strategy #

A layered approach for Nginx open source in production:

upstream app_servers {
    zone app_upstream 64k;

    # Layer 1: Primary servers — passive health check
    server 10.0.0.1:3000 max_fails=3 fail_timeout=30s;
    server 10.0.0.2:3000 max_fails=3 fail_timeout=30s;
    server 10.0.0.3:3000 max_fails=3 fail_timeout=30s;

    # Layer 2: Backup server — maintenance page
    server 10.0.0.4:8080 backup;

    keepalive 32;
}

server {
    location / {
        proxy_pass http://app_servers;

        # Retry another server on failure
        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries 3;
        proxy_next_upstream_timeout 10s;

        # Intercept backend errors for a custom error page
        proxy_intercept_errors on;
        error_page 502 503 504 = @maintenance;
    }

    location @maintenance {
        # If all backends including the backup are down
        root /var/www;
        try_files /maintenance.html =503;
        add_header Retry-After "60" always;
        add_header Cache-Control "no-store" always;
    }
}

Alerting When a Server Is Down #

#!/bin/bash
# /usr/local/bin/check-nginx-upstream.sh
# Run from cron every minute

BACKENDS=("10.0.0.1:3000" "10.0.0.2:3000" "10.0.0.3:3000")
SLACK_WEBHOOK="https://hooks.slack.com/services/xxx/yyy/zzz"

for backend in "${BACKENDS[@]}"; do
    response=$(curl -s -o /dev/null -w "%{http_code}" \
        --max-time 5 "http://$backend/health")

    if [ "$response" != "200" ]; then
        # Send an alert to Slack
        curl -s -X POST "$SLACK_WEBHOOK" \
            -H 'Content-type: application/json' \
            --data "{\"text\":\"⚠️ Backend $backend DOWN (HTTP $response)\"}"

        # Log to a file
        echo "[$(date)] Backend $backend DOWN (HTTP $response)" >> /var/log/nginx/upstream-health.log
    fi
done

# Add to crontab
# crontab -e
* * * * * /usr/local/bin/check-nginx-upstream.sh

Integration with Kubernetes and Containers #

In container environments, the Nginx health check needs to consider Kubernetes readiness and liveness probes:

# Kubernetes Deployment: make sure a pod is ready to receive traffic before adding it to the pool
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: app
          image: myapp:v2
          ports:
            - containerPort: 3000

          # Readiness probe: a pod only receives traffic if /health returns 200
          readinessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 10   # Wait 10 seconds after the container starts
            periodSeconds: 5          # Check every 5 seconds
            failureThreshold: 3       # Remove from the service after 3 failures
            successThreshold: 1       # Add back after 1 success

          # Liveness probe: restart the container on deadlock
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 30
            periodSeconds: 10
            failureThreshold: 5

If Nginx runs as an ingress in Kubernetes (e.g., the NGINX Ingress Controller), health checks are handled natively. But if Nginx stands outside the cluster as an external load balancer, you need to configure passive health checks as usual and make sure the pod’s /health endpoint is reachable.

# Nginx outside Kubernetes, load balancing to pods
# Pods are reached via NodePort or LoadBalancer Services
upstream k8s_pods {
    server k8s-node-1:30000 max_fails=3 fail_timeout=30s;  # NodePort
    server k8s-node-2:30000 max_fails=3 fail_timeout=30s;
    server k8s-node-3:30000 max_fails=3 fail_timeout=30s;

    zone k8s_upstream 64k;
    keepalive 32;
}

Testing the Health Check Endpoint #

Before deploying to production, make sure the health endpoint works correctly:

# Test the health endpoint directly to the backend
curl -v http://10.0.0.1:3000/health
# Expected:
# HTTP/1.1 200 OK
# Content-Type: application/json
# {"status":"ok","checks":{"database":"ok","cache":"ok"}}

# Simulate a database-down condition
# (turn off the database, then test)
curl -v http://10.0.0.1:3000/health
# Expected:
# HTTP/1.1 503 Service Unavailable
# {"status":"error","checks":{"database":"error: connection refused","cache":"ok"}}

# Test the health endpoint response time (should be fast, < 500ms)
for i in {1..10}; do
    time curl -s http://10.0.0.1:3000/health > /dev/null
done

# Test that the health endpoint doesn't expose sensitive information
curl http://10.0.0.1:3000/health
# Make sure there's no stack trace, version string, or credentials

# A simple monitoring script running every minute (via cron)
#!/bin/bash
# /usr/local/bin/health-monitor.sh

BACKENDS=("10.0.0.1:3000" "10.0.0.2:3000" "10.0.0.3:3000")
LOG="/var/log/nginx/health-monitor.log"
ALERT_EMAIL="[email protected]"

for backend in "${BACKENDS[@]}"; do
    result=$(curl -s -o /tmp/health_response -w "%{http_code}" \
        --max-time 5 "http://$backend/health")

    if [ "$result" = "200" ]; then
        echo "[$(date)] OK: $backend" >> $LOG
    else
        echo "[$(date)] FAIL: $backend (HTTP $result)" >> $LOG
        # Send an email alert
        echo "Backend $backend health check FAILED (HTTP $result)" | \
            mail -s "[ALERT] Backend Down: $backend" $ALERT_EMAIL

        # Or send to Slack
        curl -s -X POST https://hooks.slack.com/services/XXX \
            -H 'Content-type: application/json' \
            --data "{\"text\":\"Backend $backend DOWN (HTTP $result)\"}" \
            >> /dev/null
    fi
done

Summary #

Nginx open source only supports passive health checks — failures are detected after a client request fails, not before.
max_fails=3 fail_timeout=30s is a reasonable starting point; the default max_fails=1 fail_timeout=10s is too aggressive for most cases.
Create a health endpoint in every backend (GET /health) that checks the database, cache, and critical dependencies — respond 200 if healthy, 503 if not.
Use the zone directive in the upstream so passive health check data (fail counters) is shared across all Nginx worker processes.
For active health checks without Nginx Plus: use OpenResty + lua-resty-upstream-healthcheck, or an external cron script that probes /health.
Active health checks (proactive probing that doesn’t sacrifice client requests) are only available in Nginx Plus.
Complement with external alerting (Prometheus, cron scripts to Slack/PagerDuty) so the team knows immediately when a server is down — don’t rely only on passive detection.

← Previous: Weighted Load Balancing Next: SSL/TLS Concepts →