WebSocket Proxying #

The standard HTTP protocol is designed using a one-way request-response paradigm (the client sends a request, the server replies, and the connection closes). For modern interactive applications that require real-time data transfer — like chat applications, live stock price charts, instant notifications, or online multiplayer games — the regular HTTP protocol becomes very inefficient because it triggers high repeated polling overhead.

WebSocket solves this problem by providing a persistent full-duplex two-way communication channel over a single TCP connection. Because Nginx sits in front of our application server as a reverse proxy, Nginx must be specially configured to detect the initial WebSocket handshake, change the HTTP connection protocol into a persistent TCP tunnel, and prevent unilateral connection cuts. In this article, we’ll discuss how the upgrade handshake works, dynamically map headers, do timeout tuning, and put together an load balancing scheme for a WebSocket server cluster.

WebSocket Upgrade Handshake Workflow #

WebSocket doesn’t directly create a pure socket connection from the start. Instead, WebSocket uses the standard HTTP port (port 80 or 443) to do an initial handshake using a mechanism called HTTP Upgrade.

Here’s a visualization of the WebSocket protocol handshake sequence passing through the Nginx proxy:

sequenceDiagram
    autonumber
    actor Browser as Client Browser
    participant Nginx as Nginx Proxy
    participant Backend as Backend WS Server (Node.js/Go)

    Browser->>Nginx: GET /socket.io/ HTTP/1.1<br/>Upgrade: websocket<br/>Connection: Upgrade
    Note over Nginx: Detects the Upgrade header<br/>Maps the $connection_upgrade variable
    Nginx->>Backend: GET /socket.io/ HTTP/1.1<br/>Upgrade: websocket<br/>Connection: upgrade
    Note over Backend: Validate Key & Handshake
    Backend-->>Nginx: HTTP/1.1 101 Switching Protocols<br/>Upgrade: websocket<br/>Connection: Upgrade
    Nginx-->>Browser: HTTP/1.1 101 Switching Protocols<br/>Upgrade: websocket<br/>Connection: Upgrade
    Note over Browser, Backend: TCP Tunnel Formed (Persistent Two-Way Connection)
    Browser->>Backend: Send Data Frame (Client to Server)
    Backend->>Browser: Send Data Frame (Server to Client)

The process above happens in the following steps:

  1. The client browser sends an HTTP GET request to Nginx including two special headers: Upgrade: websocket and Connection: Upgrade. This step tells the server that the client wants to change the communication protocol.
  2. Nginx detects those headers, translates them, and forwards them to the WebSocket backend server (like a Node.js Socket.io server or a Go gorilla/websocket server).
  3. The backend server validates the handshake security key, then replies with the HTTP 101 Switching Protocols response status along with the Upgrade confirmation header.
  4. Nginx forwards that HTTP 101 response back to the client browser.
  5. The TCP connection at the lower level is kept open in memory. The communication protocol is now officially changed to an efficient persistent two-way WebSocket.

The Hardcoded Connection Header Problem (Anti-Pattern) #

To forward the WebSocket handshake, Nginx needs three important proxy directives:

proxy_http_version 1.1; # WebSockets require HTTP/1.1
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade"; # Anti-pattern if hardcoded globally!

Inserting the proxy_set_header Connection "upgrade" directive directly (hardcoded) at the global http or server block level is a very unrecommended action.

If a client sends a regular HTTP (non-WebSocket) request to our server, Nginx still inserts the Connection: upgrade header to our backend application. Some strict backend servers will be confused by receiving an upgrade header on a regular request, then refuse to process the request or return an HTTP 400 Bad Request error.

The Solution: Dynamic Connection Upgrade Mapping #

The best practice for handling WebSockets in Nginx is using the map module inside the http context (outside the server block level). We define a dynamic mapping for the Connection header based on the Upgrade header value sent by the client browser:

http {
    # If the client sends the 'Upgrade: websocket' header,
    # Nginx sets the $connection_upgrade variable value to 'upgrade'.
    # If the client doesn't send an Upgrade header (regular HTTP),
    # Nginx sets the $connection_upgrade variable value to 'close'.
    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    server {
        listen 80;
        server_name app.unisbadri.com;

        location /socket.io/ {
            proxy_pass http://websocket_backend;

            # 1. Use HTTP/1.1 (required)
            proxy_http_version 1.1;

            # 2. Forward the client's original Upgrade header
            proxy_set_header Upgrade $http_upgrade;

            # 3. Use the dynamic variable from the mapping above
            proxy_set_header Connection $connection_upgrade;

            # 4. Standard proxy headers
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}

With the mapping above, the same location block can intelligently and safely serve both regular HTTP traffic and WebSocket traffic without triggering protocol interpretation errors in the backend.

Avoiding Disconnects: WebSocket Timeout Tuning #

The biggest operational challenge after a WebSocket successfully connects at Nginx is sudden connection cuts. Clients often find their sockets disconnect every 60 seconds precisely while the page is idle (no active data transfer).

This is directly influenced by two built-in Nginx timeout directives:

  • proxy_read_timeout (default 60s): The time limit for Nginx waiting for the backend to send new data.
  • proxy_send_timeout (default 60s): The time limit for Nginx sending data to the backend.

If there’s no send-receive message activity between the client and server within 60 seconds (e.g., a user reading a chat article without typing messages), Nginx considers that TCP connection dead or hung (stale), then unilaterally closes the socket.

Solution 1: Raising the Timeout Value in Nginx #

We can raise the timeout value in the WebSocket location block to be very long (e.g., 24 hours or 86,400 seconds) so Nginx lets idle connections stay open:

location /ws/ {
    proxy_pass http://websocket_backend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    # Set the timeout limit to 24 hours
    proxy_read_timeout 86400s;
    proxy_send_timeout 86400s;
}

Solution 2: Implementing Ping-Pong (Heartbeat) on the Application Side #

Raising the timeout to 24 hours has a bad side effect: our server will be filled with “dead” socket connections that were actually abandoned by outside users but whose memory is still held by Nginx because the timeout limit hasn’t been passed.

The best step is lowering the Nginx timeout to a reasonable limit (e.g., 120s or 180s), then configuring our backend application (like Socket.io) to periodically send small lifesaver data packets (heartbeat ping-pong frames) every 25 or 50 seconds. These periodic data packets detect truly dead connections while telling Nginx that the connection is still active and must not be closed.

WebSocket Upstream Scalability (Load Balancing) #

If our WebSocket server serves tens of thousands of simultaneous connections, one backend server instance behind Nginx won’t be able to handle that workload. We must create a backend server cluster (upstream pool).

However, there’s a fundamental difference in WebSocket load balancing compared to regular HTTP: WebSockets are stateful.

After the TCP tunnel connection is established on server A, all subsequent data exchanges must keep being sent to server A. If Nginx moves the connection randomly (like using standard Round-Robin) mid-transaction, the socket connection will immediately cut and break.

We solve this connection persistence problem using the ip_hash method on the upstream block:

upstream websocket_servers {
    # ip_hash guarantees clients with the same IP are always directed to the same backend server
    ip_hash;

    server 10.0.3.5:8000;
    server 10.0.3.6:8000;
    server 10.0.3.7:8000;

    # Maintain several active keepalive connections to the backend for transfer efficiency
    keepalive 64;
}

server {
    listen 443 ssl http2;
    server_name chat.unisbadri.com;

    location /socket.io/ {
        proxy_pass http://websocket_servers;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Set the WebSocket keepalive timeout
        proxy_read_timeout 180s;
        proxy_send_timeout 180s;
    }
}

Complete Production Server Block Configuration Example (WebSocket Secure - WSS) #

Here’s a complete production-level HTTPS configuration file for deploying a trusted encrypted WebSocket server (WSS - WebSocket Secure) that is safe, fast, and protected from resource scarcity:

# 1. Dynamic Upgrade Header Mapping in the http context
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

# 2. Define the WebSocket backend server upstream pool
upstream wss_production_pool {
    ip_hash; # Use mandatory IP Hash persistence for stateful WebSocket

    server 127.0.0.1:4000 max_fails=3 fail_timeout=10s;
    server 127.0.0.1:4001 max_fails=3 fail_timeout=10s;
    server 127.0.0.1:4002 max_fails=3 fail_timeout=10s;

    keepalive 32;
}

# Automatic HTTP to HTTPS redirection
server {
    listen 80;
    server_name wss.unisbadri.com;
    return 301 https://$host$request_uri;
}

# Main HTTPS Server for WSS
server {
    listen 443 ssl http2;
    server_name wss.unisbadri.com;

    # Let's Encrypt SSL Certificate Settings
    ssl_certificate /etc/letsencrypt/live/wss.unisbadri.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/wss.unisbadri.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;

    # Network buffer optimization settings
    client_max_body_size 5m;
    client_body_buffer_size 128k;

    # Limit simultaneous connections from one IP to prevent socket spam attacks
    # (Requires a limit_conn_zone that's been defined earlier)
    # limit_conn addr 50; # Maximum 50 simultaneous socket connections per IP

    # Main route for WebSocket Proxying
    location / {
        proxy_pass http://wss_production_pool;

        # HTTP/1.1 protocol & WebSocket handshake configuration
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        # Forward the real client identity details
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeout Tuning to prevent sudden disconnects when idle
        # We set 3 minutes (180s) combined with a 30s application heartbeat
        proxy_read_timeout 180s;
        proxy_send_timeout 180s;

        # Streaming data transfer optimization
        proxy_buffering off; # Turn off buffering for interactive real-time traffic
        proxy_cache_bypass $http_upgrade;
    }

    # Logging configuration
    access_log /var/log/nginx/wss-access.log;
    error_log /var/log/nginx/wss-error.log warn;
}

OS & Nginx Tuning for WebSocket Scalability (100k+ Connections) #

WebSockets are persistent and long-lived, meaning the TCP connections between client-Nginx and Nginx-backend stay open simultaneously for long periods. At high traffic, our server can quickly run out of file descriptor capacity if the default Linux OS and Nginx limits aren’t adjusted.

To support hundreds of thousands of simultaneous WebSocket connections, we must do tuning on both the Linux operating system configuration and our Nginx files:

1. Raising the Linux File Descriptor Limit (limits.conf) #

By default, Linux limits the number of file descriptors per user to 1024. Because every active TCP connection is represented as one file descriptor in the OS, our server can’t serve more than 1024 simultaneous connections before triggering the “Too many open files” error.

Edit the /etc/security/limits.conf file to raise the limit for the nginx user (or the system user running Nginx):

nginx       soft    nofile          100000
nginx       hard    nofile          100000

2. Linux Kernel Tuning (sysctl.conf) #

We also need to adjust kernel settings in /etc/sysctl.conf so the OS can manage TCP connection memory efficiently:

# Increase the maximum TCP connection backlog queue
net.core.somaxconn = 65535

# Increase the overall system maximum file descriptor limit
fs.file-max = 2097152

# Expand the local port range for outbound connections to the backend
net.ipv4.ip_local_port_range = 1024 65535

# Enable TIME_WAIT socket reuse for fast connections
net.ipv4.tcp_tw_reuse = 1

Run the sudo sysctl -p command to immediately apply the kernel parameter changes above without needing to reboot the server.

3. Adjustment Configuration in nginx.conf #

Inside the main nginx.conf configuration file, we must raise the worker connections capacity to align with the file descriptor limit in the OS:

# Main configuration in nginx.conf (global context)
worker_processes auto;
worker_rlimit_nofile 100000; # Must align with limits.conf

events {
    worker_connections 50000; # Maximum connections per worker process
    use epoll;                # High-performance I/O multiplexing method for Linux
    multi_accept on;          # Accept all new connections at once if possible
}

With adjustments at those three layers (OS Limits, Kernel parameters, and the Nginx event model), our server is ready to handle tens of thousands of active WebSocket connections simultaneously without experiencing socket overload or memory leak obstacles.

Summary and Best Practices #

  • Use the connection_upgrade map: Avoid writing Connection "upgrade" permanently at the global level. Use the map mapping block to flexibly detect the client request type.
  • Turn Off proxy_buffering: For real-time WebSocket traffic, always set proxy_buffering off; so Nginx doesn’t hold data frames in buffer memory and immediately forwards them to the client instantly.
  • Apply ip_hash for Stateful Upstreams: WebSockets are persistent on one physical server. Use the ip_hash load balancing method so client socket connections don’t break from moving between backend server instances.
  • Combine Nginx Timeouts with Application Heartbeats: Setting Nginx timeouts too long wastes server memory. Use a reasonable limit (e.g., 180 seconds) combined with heartbeat data sending in our client-server application code.

← Previous: Single Page Application (React/Vue)   Next: Common Errors →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact