Config Debugging #

Debugging Nginx configuration can be a confusing process if we don’t know where to start. Nginx often doesn’t give error messages that directly point to the logic mistake inside the configuration file. For example, our server could run smoothly without syntax errors, but client requests mysteriously fall into the wrong server block, or an unexpected location block, or the proxy headers we created turn out not to be forwarded to the backend.

These kinds of logic obstacles require structured configuration tracing and debugging techniques. In this article, we’ll discuss how to dissect Nginx’s block selection logic, map routes using diagnostic commands, inspect runtime variables directly, and apply problem isolation with a systematic method.

Location Block Selection Priority Rules #

One of the biggest sources of confusion when debugging Nginx configuration is understanding which location block wins the route match when there are several overlapping location blocks.

Nginx doesn’t match location blocks based on their written order from top to bottom inside the configuration file, but follows this matching operator priority order:

  1. Exact Match (=): Matches the URI exactly character by character. If there’s a match, the search immediately stops and this location is used.
  2. Preferential Prefix (^~): Matches the longest URI prefix. If the longest match is found under this operator, Nginx directly uses this route and ignores regular expression checks.
  3. Regex Match (~ or ~*): Matches regular expressions case-sensitively (~) or case-insensitively (~*). Nginx scans from top to bottom and the first matching regex route wins.
  4. Standard Prefix (no operator): Matches the longest URI prefix. This route is only used if no regular expression matches.

Here’s the location block selection decision flow in Nginx:

flowchart TD
    Req["Request URI Arrives"] --> MatchExact{"1. Check Exact Match '='?"}
    MatchExact -->|"Yes (Matches)"| Exact["Use Location '='<br/>(Stop the Search)"]
    MatchExact -->|"No"| Preferential{"2. Check Preferential Prefix '^~'?"}
    
    Preferential -->|"Yes (Longest Match)"| PreferentialAction["Use Location '^~'<br/>(Stop the Search, Ignore Regex)"]
    Preferential -->|"No"| RegexCheck{"3. Check Regex Match '~' or '~*'?"}
    
    RegexCheck -->|"Something Matches"| RegexAction["Use the First Matching Regex<br/>(Stop the Search)"]
    RegexCheck -->|"Nothing Matches"| StandardPrefix{"4. Check Standard Prefix?"}
    
    StandardPrefix -->|"Something Matches"| StandardAction["Use the Longest Matching Prefix"]
    StandardPrefix -->|"Nothing"| DefaultRoute["5. Fallback to the Default Location '/'"]

    classDef default fill:#f9f9f9,stroke:#d1d5db,stroke-width:1px,color:#111827;
    classDef matchStyle fill:#eff6ff,stroke:#3b82f6,stroke-width:2px,color:#1e3a8a;
    classDef stopStyle fill:#f0fdf4,stroke:#15803d,stroke-width:2px,color:#166534;
    class MatchExact,Preferential,RegexCheck,StandardPrefix matchStyle;
    class Exact,PreferentialAction,RegexAction,StandardAction,DefaultRoute stopStyle;

First Step: Syntax Validation with nginx -t #

The first step in every debugging scenario is validating the configuration file syntax. We must not reload the configuration on a production server before making sure there are no typos or missing punctuation.

# Test the default configuration (/etc/nginx/nginx.conf)
sudo nginx -t

# Expected successful output:
# nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
# nginx: configuration file /etc/nginx/nginx.conf test is successful

If an error occurs, nginx -t will immediately show the file line and error type:

nginx: [emerg] unexpected ";" in /etc/nginx/conf.d/app.conf:15
nginx: configuration file /etc/nginx/nginx.conf test failed

Testing an Alternative Configuration File #

When we’re experimenting with a new configuration and don’t want to disturb the running server’s main configuration file, we can duplicate the configuration to a backup folder then validate it specifically:

sudo nginx -t -c /var/www/test-env/nginx.conf

Dumping the Active Configuration with nginx -T #

On production servers with dozens of virtual host configuration files interconnected using the include directive, tracking which directive is active or overridden can be very tiring.

We can use the nginx -T command (capital T) to instruct Nginx to merge all include files into one full text dump output in the terminal.

# Display the entire active configuration in the terminal
sudo nginx -T

Because the output can be very long, we can combine it with the grep search utility to verify whether a certain directive has been loaded correctly:

# Find where a certain server_name declaration is located
sudo nginx -T | grep -A 10 "server_name app.unisbadri.com"

# Check whether SSL parameters have been included everywhere
sudo nginx -T | grep -E "ssl_protocols|ssl_ciphers"

# Count the number of active server blocks read by Nginx
sudo nginx -T | grep -c "server_name"

Debugging with return: Tracing Location Mapping #

When we’re confused about why a request is processed incorrectly by Nginx (e.g., static JS files return a 404 error even though the file exists on disk), we can use the tactic of inserting a temporary return directive with a plain text response to detect which location block wins the route match.

server {
    listen 80;
    server_name test.unisbadri.com;

    location / {
        return 200 "Selected Route: Root Location ()\n";
    }

    location /api/ {
        return 200 "Selected Route: API PREFIX Location\n";
    }

    location ^~ /api/v2/ {
        return 200 "Selected Route: API V2 PREFERENTIAL Location\n";
    }

    location ~* \.php$ {
        return 200 "Selected Route: PHP REGEX Location\n";
    }
}

After reloading that configuration on the test server, we send requests using curl to see the response text:

# Case 1: Request to the root path
curl http://test.unisbadri.com/
# Output: Selected Route: Root Location ()

# Case 2: Request to /api/users
curl http://test.unisbadri.com/api/users
# Output: Selected Route: API PREFIX Location

# Case 3: Request to /api/v2/auth (preferential wins the route from regex)
curl http://test.unisbadri.com/api/v2/auth.php
# Output: Selected Route: API V2 PREFERENTIAL Location

This simple tactic drastically cuts debugging time because we don’t need to guess which regular expression rule was mistyped.

Debugging with add_header: Runtime Variable Inspection #

Nginx has dozens of internal variables (like $uri, $request_uri, $args, $upstream_addr, $upstream_response_time, etc.) whose values change dynamically with every request.

To find out the contents of those variable values directly while a request is being processed, we can add custom response headers using the add_header directive with the always flag:

server {
    listen 80;
    server_name debug.unisbadri.com;

    location / {
        proxy_pass http://nodejs_backend;

        # Add debug headers to the HTTP response
        add_header X-Debug-URI "$uri" always;
        add_header X-Debug-Request-URI "$request_uri" always;
        add_header X-Debug-Args "$args" always;
        add_header X-Debug-Upstream-Addr "$upstream_addr" always;
        add_header X-Debug-Upstream-Response "$upstream_response_time" always;
        add_header X-Debug-Client-IP "$remote_addr" always;
    }
}

Send requests using the -I flag on curl to inspect the response headers coming back from the server:

curl -I "http://debug.unisbadri.com/api/users?status=active"

# HTTP/1.1 200 OK
# Server: nginx
# X-Debug-URI: /api/users
# X-Debug-Request-URI: /api/users?status=active
# X-Debug-Args: status=active
# X-Debug-Upstream-Addr: 127.0.0.1:3000
# X-Debug-Upstream-Response: 0.045
# X-Debug-Client-IP: 203.0.113.50

[!WARNING] Never leave this custom debug header configuration active permanently in a production environment. It can expose internal network architecture information, backend IP addresses, and our port structure to outside parties who could misuse it for attacks.

debug_connection: Deep Per-IP Tracing #

Turning on the debug log level globally on a production server is a dangerous action because the massive debug log writing volume will eat gigabytes of disk space in minutes, while also burdening the server’s CPU I/O performance.

Nginx provides a clever solution called the debug_connection directive inside the events context. This directive lets us turn on high-level debug log recording exclusively only for certain IP addresses or subnets (e.g., our developer’s office laptop IP).

# In the main context (the very top of nginx.conf, outside http{})
error_log /var/log/nginx/error.log warn; # The global default level stays safe

events {
    worker_connections 1024;

    # Enable debug log recording only for our tester's IP address
    debug_connection 203.0.113.42;
    
    # We can also include an internal VPN subnet segment
    debug_connection 10.8.0.0/24;
}

When a laptop with the IP 203.0.113.42 sends a request to our server, Nginx writes the internal processing logs in great detail to /var/log/nginx/error.log (like header parsing processes, regex matching, rewrite phases, SSL handshakes, to byte transfer), while requests from other internet users’ IPs are still recorded using the standard warn log level.

Debugging Rewrite Rules with rewrite_log #

When we use the rewrite directive to dynamically change URLs (e.g., pointing user-friendly URLs to internal files), tracking whether the rewrite regex matching runs correctly or experiences an infinite loop is very hard to do just by looking at the browser.

Nginx provides a built-in module that can record every rewrite rule evaluation process into the error log file using the rewrite_log directive.

  1. Enable rewrite_log at the server block level:

    server {
    listen 80;
    server_name app.unisbadri.com;

    # Enable rewrite rule evaluation recording
    rewrite_log on;

    # We must set error_log to the 'notice' level so rewrite logs are recorded
    error_log /var/log/nginx/app-error.log notice;

    rewrite ^/users/([0-9]+)/?$ /profile.php?id=$1 last;
    rewrite ^/posts/([a-zA-Z0-9\-]+)/?$ /article.php?slug=$1 last;
}

  2. Analyze the Rewrite Log Output: When a /users/42 request comes in, Nginx writes the regex matching stages into the log:

    2026/06/16 14:30:00 [notice] 12345#12345: *127 "^/users/([0-9]+)/?$" matches "/users/42", rewriting to "/profile.php?id=42", client: 203.0.113.50, server: app.unisbadri.com, request: "GET /users/42 HTTP/1.1", host: "app.unisbadri.com"

    If a regex error occurs so the request isn’t rewritten, the log shows:

    2026/06/16 14:30:05 [notice] 12345#12345: *128 "^/users/([0-9]+)/?$" does not match "/users/abc", client: 203.0.113.50, server: app.unisbadri.com, request: "GET /users/abc HTTP/1.1", host: "app.unisbadri.com"

    If we mistyped the last or break flag triggering an endless loop (Nginx limits internal redirects to a maximum of 10 rounds), Nginx returns a 500 error and writes the following log:

    2026/06/16 14:30:10 [error] 12345#12345: *129 rewrite or internal redirection cycle while processing "/profile.php", ...

    By enabling rewrite_log on and raising the log level to notice, we can precisely dismantle internal redirect problems.

Diagnosing Server Block Selection (Virtual Host Mismatch) #

Another confusing scenario is when we have several server block files with different server_name values, but requests to the b.com domain are always answered by the configuration belonging to a.com.

To solve the server block selection (virtual host resolution) problem, we can apply the following diagnosis steps:

  1. Check the default_server Directive: Nginx uses the server block with the default_server parameter on the listen directive if the request domain doesn’t match any server_name. If no server block is declared as default_server, Nginx automatically points to the first server block loaded in memory as the default server.

  2. Check the File Loading Order: If we use the include /etc/nginx/conf.d/*.conf; directive, the OS loads files based on the alphabetical order of their file names. The a.com.conf file loads before b.com.conf. If b.com is mistyped in its configuration (e.g., missing one letter), requests to b.com will be answered by the default server (which is a.com.conf because it loads first).

  3. Test Resolution with Curl: We can send requests by forcing a custom HTTP Host header to simulate server name matching:

    # Test a request to the server IP by forcing the Host of domain B
curl -H "Host: b.com" http://OUR_SERVER_IP/

    Combine this with the return 200 "server b active" tactic inside the server block to verify directly.

Problem Isolation with the Binary Search Approach #

When we face a very complex virtual host configuration (e.g., hundreds of lines of code with dozens of rewrite rules, location blocks, header manipulation, and cache bypass) that triggers anomalous server behavior, debugging by guessing which line is wrong often ends in failure.

We can apply a configuration Binary Search strategy to localize the problem:

  1. Duplicate the file: Duplicate the problematic virtual host configuration file to a backup file.
  2. Comment Out Half the Block: Disable (put a # at the start of the line) about half of the dynamic configuration block count (e.g., disable the entire caching and compression module).
  3. Validate and Reload: Run nginx -t then reload the server.
  4. Test the Behavior: Send a test request.
    • If the problem disappears: The problematic directive is inside the half of the configuration block we just commented out.
    • If the problem remains: The problematic directive is inside the still-active half of the configuration.
  5. Narrow the Area: Repeat dividing that active half until we successfully isolate exactly one single directive that’s the culprit.

This systematic elimination strategy guarantees finding the root cause logically without being influenced by our subjective assumptions.

Verifying a Successful Graceful Reload #

After we make configuration changes and run sudo systemctl reload nginx, how can we be sure that Nginx really reloaded the new configuration successfully without hidden failures?

We can verify it through the following steps:

  • Step 1: Check the systemd journal logs to see the Nginx reload signal status:
    sudo journalctl -u nginx --since "10 minutes ago"

# Expected successful reload log:
# systemd[1]: Reloading nginx - high performance web server...
# nginx[12345]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
# nginx[12345]: nginx: configuration file /etc/nginx/nginx.conf test is successful
# systemd[1]: Reloaded nginx - high performance web server.
  • Step 2: Check the master process Process ID (PID) number of Nginx. During a graceful reload, the master process PID doesn’t change, only the worker process PIDs are gradually replaced:
    # Note the Nginx master PID
cat /run/nginx.pid

# Check the creation time of new worker processes
ps -eo pid,ppid,lstart,cmd | grep nginx
    If the new worker process creation time aligns with the second we ran the reload command, the graceful reload transition has run successfully on our server.

Debugging Technique Summary #

  • nginx -T: Dump the entire contents of the integrated active configuration files to track if there are duplicate directives accidentally overridden.
  • Use a temporary return: Detect which location block route wins the URI match by returning plain text responses.
  • add_header X-Debug-*: Display internal runtime variable values like $upstream_addr to track backend request load distribution.
  • debug_connection: Limit verbose debug logging activation only to our laptop’s IP to preserve production server performance.
  • Apply Binary Search: Find the problematic configuration line in giant configuration files by periodically disabling half of the section.

← Previous: Common Errors   Next: Diagnostic Tools →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact