Round Robin #

Round robin is Nginx’s default load balancing algorithm — if you don’t specify another algorithm in an upstream block, this is what gets used. It works simply: each request is forwarded to the next server in the list, in rotation. This simplicity makes it the right starting point for understanding load balancing in Nginx.

How It Works Internally #

Nginx maintains a pointer that points to the “next” server in the upstream pool. Each time a request arrives, the pointer moves to the next server. When it reaches the end of the list, the pointer wraps back to the start.

sequenceDiagram
    participant C as Client
    participant N as Nginx
    participant A as Server A
    participant B as Server B
    participant CC as Server C

    C->>N: Request 1
    N->>A: Forward to Server A
    A->>N: Response
    N->>C: Response to client

    C->>N: Request 2
    N->>B: Forward to Server B
    B->>N: Response
    N->>C: Response to client

    C->>N: Request 3
    N->>CC: Forward to Server C
    CC->>N: Response
    N->>C: Response to client

    C->>N: Request 4
    N->>A: Back to Server A
    A->>N: Response
    N->>C: Response to client

Each server gets a roughly equal number of requests over time. This assumes every request has similar processing load and every server has equal capacity — assumptions that hold for many cases, but not all.

Basic Configuration #

# /etc/nginx/conf.d/myapp.conf

# Upstream pool definition — the list of backend servers
upstream app_servers {
    server 10.0.0.1:3000;
    server 10.0.0.2:3000;
    server 10.0.0.3:3000;
}

server {
    listen 443 ssl;
    http2  on;
    server_name example.com;

    location / {
        proxy_pass         http://app_servers;
        proxy_http_version 1.1;
        proxy_set_header   Connection    "";
        proxy_set_header   Host          $host;
        proxy_set_header   X-Real-IP     $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
    }
}

The upstream name (app_servers) can be any valid identifier — it’s only used as a reference in proxy_pass. Nginx distributes requests to all three servers in rotation.

Server Parameters: Controlling Individual Behavior #

Each server in an upstream block can be configured with extra parameters controlling how it’s treated in the rotation:

upstream app_servers {
    # Normal server — participates in the full rotation
    server 10.0.0.1:3000;

    # Server temporarily removed from rotation manually
    # Requests won't be sent here, but the server is still counted
    # in hashing (important for ip_hash so the mapping isn't scrambled)
    server 10.0.0.2:3000 down;

    # Backup server — only active if all main servers are down/unavailable
    # Ideal for showing a maintenance page
    server 10.0.0.3:3000 backup;

    # max_fails=3: after 3 failures within the fail_timeout window...
    # fail_timeout=30s: ...for 30 seconds, the server is marked unavailable
    #                   after 30 seconds, Nginx tries it again (1 request)
    server 10.0.0.4:3000 max_fails=3 fail_timeout=30s;

    # max_conns=100: limit the number of simultaneous connections to this server
    # useful for protecting a weaker server from overload
    server 10.0.0.5:3000 max_conns=100;

    # Combination of all parameters:
    server 10.0.0.6:3000 max_fails=3 fail_timeout=30s max_conns=200;
}

How max_fails and fail_timeout Work #

flowchart TD
    REQ["Request arrives"] --> CHECK{"Is Server A\navailable?"}
    CHECK -- "Yes" --> SEND["Send to Server A"]
    SEND --> RESULT{"Success?"}
    RESULT -- "Yes" --> OK["Response to client\nReset fail counter"]
    RESULT -- "No\nerror or timeout" --> FAIL["Increment fail counter\nTry another server via\nproxy_next_upstream"]
    FAIL --> MAX{"Fail counter\n≥ max_fails?"}
    MAX -- "No" --> NEXT["Server A still\nin rotation"]
    MAX -- "Yes" --> DOWN["Mark Server A 'down'\nfor fail_timeout seconds"]
    DOWN --> WAIT["After fail_timeout:\nTry 1 request to Server A"]
    WAIT --> RESULT2{"Success?"}
    RESULT2 -- "Yes" --> RESTORE["Server A returns\nto normal rotation"]
    RESULT2 -- "No" --> DOWN

Concretely:

max_fails=3 fail_timeout=30s:

[t=0s]  1st failure → fail counter = 1
[t=5s]  2nd failure → fail counter = 2
[t=10s] 3rd failure → fail counter = 3 → Server MARKED DOWN
[t=40s] (30s elapsed) → Nginx sends 1 request to the server
[t=40s] Success → Server back in rotation, fail counter = 0
[t=40s] Failed → Server marked down again, wait another 30s

Default values if not specified: max_fails=1 fail_timeout=10s — a single failure takes the server out for 10 seconds. That’s too aggressive for most cases, especially if a server is occasionally slow due to GC pauses.

Failover: proxy_next_upstream #

When Nginx fails to forward a request to the selected server, it can try the next one. This behavior is controlled by proxy_next_upstream:

upstream app_servers {
    server 10.0.0.1:3000 max_fails=2 fail_timeout=10s;
    server 10.0.0.2:3000 max_fails=2 fail_timeout=10s;
    server 10.0.0.3:3000 max_fails=2 fail_timeout=10s;
}

server {
    location / {
        proxy_pass http://app_servers;

        # Conditions that trigger a retry to another server
        # error: connection failed/refused
        # timeout: proxy_connect_timeout or proxy_read_timeout exceeded
        # http_502, http_503, http_504: errors from the backend
        proxy_next_upstream error timeout http_502 http_503 http_504;

        # Total time limit for all attempts (all servers)
        proxy_next_upstream_timeout 10s;

        # Maximum number of attempts on other servers
        proxy_next_upstream_tries 3;
    }
}
Be careful with POST and data-mutating operations. proxy_next_upstream will resend the request to another server if the first server errors. If the backend already started processing the POST request before sending the error, the request could be executed twice (double write, double transaction). Use proxy_next_upstream carefully for non-idempotent endpoints.

Backup Servers and Maintenance Pages #

Backup servers only receive traffic when all main servers are unavailable. This is an elegant way to show a maintenance page instead of a 502 error:

upstream app_servers {
    server 10.0.0.1:3000 max_fails=3 fail_timeout=30s;
    server 10.0.0.2:3000 max_fails=3 fail_timeout=30s;

    # Backup server: a static Nginx serving a maintenance page
    server 10.0.0.3:8080 backup;
}

server {
    # Backup server: run a separate Nginx on port 8080
    # that only serves a static maintenance page
    listen 8080;
    server_name _;

    root /var/www/maintenance;

    location / {
        try_files /maintenance.html =503;
        add_header Retry-After "300" always;
    }
}

Or more simply, use error_page together with proxy_intercept_errors:

server {
    location / {
        proxy_pass http://app_servers;
        proxy_intercept_errors on;

        # Show a maintenance page if all backends are down (502/503)
        error_page 502 503 504 /maintenance.html;
    }

    location = /maintenance.html {
        root /var/www;
        internal;
        add_header Content-Type text/html;
    }
}

Keepalive Pool to Upstream #

For high performance, Nginx can maintain a pool of keepalive connections to backends rather than opening a new connection for every request:

upstream app_servers {
    server 10.0.0.1:3000;
    server 10.0.0.2:3000;
    server 10.0.0.3:3000;

    # Maintain 32 idle connections to each server
    # (total pool = 32 per server, not 32 overall)
    keepalive 32;

    # How long idle connections are maintained
    keepalive_timeout 60s;

    # How many requests per connection before closing and reopening
    keepalive_requests 1000;
}

server {
    location / {
        proxy_pass         http://app_servers;
        proxy_http_version 1.1;   # Required for keepalive
        proxy_set_header   Connection "";  # Remove the Connection header from the client
    }
}

The real impact of keepalive: for an app with 1000 req/s to a local backend, the TCP handshake overhead (~1ms per connection) times 1000 = 1 second of overhead per second just from handshakes. With keepalive, this overhead approaches zero.

Shared Memory Zones: The zone Directive #

Without the zone directive, each Nginx worker process has its own upstream data — fail counters aren’t shared between workers. With zone, the data is shared in shared memory:

upstream app_servers {
    # Create a shared memory zone named "app" of 64k
    # Fail counter data, server status, etc. are shared across all workers
    zone app_upstream 64k;

    server 10.0.0.1:3000;
    server 10.0.0.2:3000;
    server 10.0.0.3:3000;

    keepalive 32;
}

With zone, when worker-1 records a failure for Server A, worker-2 and worker-3 know about it immediately. This makes the passive health check far more responsive — no need for every worker to experience the failure itself before taking the server out of rotation.

Logging and Monitoring Round Robin #

http {
    # Log format including backend info
    log_format upstream_log
        '$remote_addr [$time_local] "$request" '
        '$status $upstream_addr '          # selected backend IP
        '$upstream_status '                # status code from the backend
        '$upstream_response_time '         # time the backend took to respond
        '$request_time';

    server {
        access_log /var/log/nginx/app-access.log upstream_log;

        location / {
            proxy_pass http://app_servers;
        }
    }
}

# Analyze traffic distribution per server
awk '{print $5}' /var/log/nginx/app-access.log | sort | uniq -c | sort -rn
# Output:
#  3421 10.0.0.1:3000
#  3389 10.0.0.2:3000
#  3398 10.0.0.3:3000
# → Even distribution: round robin is working well

# Monitor errors per server
awk '{print $5, $6}' /var/log/nginx/app-access.log | grep "50[0-9]" | sort | uniq -c
# → Detect which server errors often

Decision Tree: When to Use Round Robin #

flowchart TD
    START["Start choosing a\nload balancing algorithm"] --> Q1{"Are all servers\nequal in capacity?"}
    Q1 -- "No" --> WEIGHTED["Use Weighted\n(according to capacity)"]
    Q1 -- "Yes" --> Q2{"Do request durations\nvary widely?"}
    Q2 -- "Yes (some take\nseconds/minutes)" --> LEASTCONN["Use Least Connections\n(adaptive to load)"]
    Q2 -- "No\n(consistently < 500ms)" --> Q3{"Need session\npersistence?"}
    Q3 -- "Yes" --> Q4{"Session in local\nserver memory?"}
    Q4 -- "Yes" --> IPHASH["Use IP Hash\nor cookie hash"]
    Q4 -- "No\nRedis or DB" --> RR["✓ Use Round Robin\noptimal for this case"]
    Q3 -- "No" --> RR

Managing Upstream Without Downtime #

Changes to the upstream block can be applied without cutting in-flight connections using nginx -s reload:

# Safe reload workflow:

# 1. Edit the configuration
vim /etc/nginx/conf.d/upstream.conf

# 2. Validate the configuration before applying
nginx -t
# nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
# nginx: configuration file /etc/nginx/nginx.conf test is successful

# 3. Reload: apply the new configuration without shutting down the server
nginx -s reload
# Old worker processes finish existing connections
# New worker processes start with the new configuration
# No requests are interrupted

Internally, when nginx -s reload runs:

  1. The master process reads the new configuration
  2. The master process forks new worker processes with the new configuration
  3. Old worker processes are signaled not to accept new requests
  4. Old worker processes finish in-flight requests
  5. Old worker processes finish and exit gracefully

Already-established keepalive connections to clients are closed after keepalive_timeout — no connection is force-dropped.

Adding a Server to Upstream (Zero-Downtime) #

# Before:
upstream app_servers {
    server 10.0.0.1:3000;
    server 10.0.0.2:3000;
    keepalive 32;
    zone app_upstream 64k;
}

# After (add a new server):
upstream app_servers {
    server 10.0.0.1:3000;
    server 10.0.0.2:3000;
    server 10.0.0.3:3000;  # ← add this line
    keepalive 32;
    zone app_upstream 64k;
}
# nginx -t && nginx -s reload

Removing a Server from Upstream (Graceful Drain) #

# Step 1: Mark it as down (stop new requests)
upstream app_servers {
    server 10.0.0.1:3000;
    server 10.0.0.2:3000 down;  # ← no longer accepts new requests
    keepalive 32;
    zone app_upstream 64k;
}
# nginx -t && nginx -s reload

# Wait for in-flight requests to finish (check the logs)
# tail -f /var/log/nginx/access.log | grep "10\.0\.0\.2"

# Step 2: Remove from the configuration once you're sure there are no active requests
upstream app_servers {
    server 10.0.0.1:3000;
    # server 10.0.0.2:3000;  ← can now be removed
    keepalive 32;
    zone app_upstream 64k;
}
# nginx -t && nginx -s reload

Tuning Nginx for High Throughput with Round Robin #

For high-traffic production, several Nginx settings need adjustment so load balancing runs optimally:

# /etc/nginx/nginx.conf
worker_processes auto;         # One worker per CPU core
worker_rlimit_nofile 65535;    # File descriptor limit per worker

events {
    worker_connections 4096;   # Maximum simultaneous connections per worker
    use epoll;                 # Efficient event model on Linux
    multi_accept on;           # Accept multiple connections at once
}

http {
    upstream app_servers {
        server 10.0.0.1:3000;
        server 10.0.0.2:3000;
        server 10.0.0.3:3000;
        keepalive 64;
        keepalive_requests 10000;
        zone app_upstream 128k;
    }

    # Disable access logs on very frequently hit endpoints
    # (e.g., health checks called every second)
    location /health {
        proxy_pass http://app_servers;
        access_log off;
    }

    # Enable sendfile for serving static files
    sendfile    on;
    tcp_nopush  on;
    tcp_nodelay on;
}

Calculating Theoretical Capacity #

Nginx maximum capacity (theoretical):
  worker_processes = 4 (4 cores)
  worker_connections = 4096

  Total simultaneous connections = 4 × 4096 = 16,384
  (including client connections AND backend connections)

  For a reverse proxy, each request uses 2 connections
  (client → Nginx, Nginx → backend)

  Total simultaneous requests = 16,384 / 2 = 8,192

With a keepalive pool (backend connections reused):
  Total simultaneous requests is much higher
  because backend connections aren't opened/closed for every request

Common Mistakes to Avoid #

# ✗ WRONG: No zone directive — fail counters aren't synchronized between workers
upstream app_servers {
    server 10.0.0.1:3000 max_fails=3 fail_timeout=30s;
    server 10.0.0.2:3000 max_fails=3 fail_timeout=30s;
    # No zone!
}
# Different worker processes each have their own fail counters.
# It takes max_fails × number_of_workers failures before a server is marked down

# ✓ CORRECT: With the zone directive
upstream app_servers {
    zone app_upstream 64k;  # Shared memory across all workers
    server 10.0.0.1:3000 max_fails=3 fail_timeout=30s;
    server 10.0.0.2:3000 max_fails=3 fail_timeout=30s;
}

# ✗ WRONG: Forgetting proxy_http_version 1.1 when using keepalive
upstream app_servers {
    server 10.0.0.1:3000;
    keepalive 32;
}
server {
    location / {
        proxy_pass http://app_servers;
        # No proxy_http_version 1.1!
        # Keepalive won't work with HTTP/1.0 (default)
    }
}

# ✓ CORRECT: Always include proxy_http_version 1.1 when using keepalive
server {
    location / {
        proxy_pass         http://app_servers;
        proxy_http_version 1.1;     # ← Required for keepalive
        proxy_set_header   Connection ""; # ← Remove the Connection header from the client
    }
}

Summary #

  • Round robin is Nginx’s default algorithm — no extra directive needed, just list the servers in an upstream block.
  • The max_fails and fail_timeout parameters control passive health checks — the default max_fails=1 fail_timeout=10s is often too aggressive; set max_fails=3 fail_timeout=30s as a more reasonable starting point.
  • backup servers only activate when all main servers are down — ideal for an informative maintenance page.
  • proxy_next_upstream allows retries to another server, but be careful with non-idempotent POST requests — they can cause double processing.
  • Use the zone directive so fail counter and server status data are shared across all Nginx worker processes.
  • Add keepalive in the upstream block to avoid repeated TCP handshake overhead to the backend.

← Previous: Proxy Cache   Next: Least Connections →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact