Diagnostic Tools #

Effectively diagnosing Nginx and its supporting network problems requires a deep understanding of the right tools. Some of these diagnostic tools are installed by default in almost every Linux operating system distribution, while others are third-party utilities we can install as needed.

Knowing the use of each tool, understanding its command parameters, and knowing when to use them are important skills equal in value to understanding Nginx configuration syntax itself. In this article, we’ll discuss how to use various diagnostic tools to analyze HTTP connections, inspect SSL, monitor network sockets, trace system calls, and set up continuous monitoring.

Diagnostic Tool Correlation in the Request-Response Cycle #

When isolating a problem, we must track at which point the failure occurs along the request and response cycle. Here’s a flow diagram showing the correlation of diagnostic tools at each Nginx communication layer:

flowchart LR
    Client["1. Client Browser / App"] -->|"curl -v / openssl s_client"| NginxSSL["2. Nginx SSL Termination"]
    NginxSSL -->|"ss -tlnp / netstat"| NginxSocket["3. TCP Sockets"]
    NginxSocket -->|"strace / sysctl"| NginxOS["4. OS Kernel / Syscalls"]
    NginxOS -->|"GoAccess / access.log"| Backend["5. Backend Server (Node.js/PHP)"]

    classDef default fill:#f9f9f9,stroke:#d1d5db,stroke-width:1px,color:#111827;
    classDef toolStyle fill:#fffbeb,stroke:#d97706,stroke-width:2px,color:#78350f;
    class Client,NginxSSL,NginxSocket,NginxOS,Backend toolStyle;

curl: The HTTP Swiss Army Knife #

The first tool we must use when a web access obstacle occurs is curl. This CLI utility lets us make HTTP/HTTPS requests precisely without being affected by local browser cache storage or JavaScript.

1. Basic Request with Verbose Output #

Use the -v (verbose) flag to see the entire HTTP conversation between the client and Nginx, including the DNS resolution process, TLS handshake, request headers sent by the browser, and response headers from the server.

curl -v https://app.unisbadri.com/

2. Reading Only the Response Headers #

Use the -I (Capital i) flag to do a HEAD request to see the response header summary without downloading the entire page body content (very useful for checking Cache-Control, Server tokens, or Security Headers):

curl -I https://app.unisbadri.com/

3. Testing a Domain Without DNS Changes (DNS Spoofing) #

If we just deployed a new HTTPS virtual host configuration on the server and the domain hasn’t publicly pointed to our server IP yet, we can force curl to ignore global DNS and directly hit our server IP for that domain using the --resolve flag:

# Format: --resolve DOMAIN:PORT:IP_SERVER
curl -v --resolve app.unisbadri.com:443:203.0.113.10 https://app.unisbadri.com/

4. Tracking HTTP Processing Times (Profiling) #

We can create an internal profiling script to map how many milliseconds are spent on DNS lookup, TCP connection, SSL negotiation, first response waiting time (TTFB), and total download time:

curl -o /dev/null -s -w \
    "DNS Lookup: %{time_namelookup}s\nTCP Connect: %{time_connect}s\nSSL Handshake: %{time_appconnect}s\nTTFB (First Byte): %{time_starttransfer}s\nTotal Time: %{time_total}s\n" \
    https://app.unisbadri.com/

openssl s_client: SSL/TLS Debugging #

When curl returns an SSL connection error (like SSL_ERROR_SYSCALL), we must switch to using the openssl s_client utility to do in-depth inspection at the encryption protocol level.

1. Checking the Certificate and Chain of Trust #

Nginx must return the domain certificate along with the intermediate CA certificate completely. We can verify it using the command:

echo | openssl s_client -connect app.unisbadri.com:443 -servername app.unisbadri.com -showcerts

Pay attention to the Certificate chain list. If the chain is broken, client browsers will show an insecure warning even though our domain certificate is valid.

2. Testing Specific TLS Protocols #

To verify whether our Nginx server has correctly disabled outdated protocols according to security hardening rules, we can force connections using specific TLS versions:

# Test whether the server accepts TLS 1.0 (Should be rejected / Connection refused)
echo | openssl s_client -connect app.unisbadri.com:443 -tls1

# Test whether the server accepts TLS 1.2 (Should be accepted)
echo | openssl s_client -connect app.unisbadri.com:443 -tls1_2

# Test whether the server accepts TLS 1.3 (Should be accepted)
echo | openssl s_client -connect app.unisbadri.com:443 -tls1_3

ss and netstat: TCP Connection Inspection #

To find out whether the problem is at the network layer (like blocked ports, full socket queues, or running out of local ports), we use the ss utility (the modern replacement for netstat).

1. Checking All Listening Ports #

The initial step to detect port conflicts is seeing which processes are currently listening for incoming traffic:

sudo ss -tlnp
# -t: TCP, -l: Listening, -n: Numeric (show port numbers), -p: Show the process name

2. Analyzing the TCP Connection State Distribution #

TCP socket traffic goes through various lifecycle stages. We can count how many of our Nginx connections are in each state:

sudo ss -tn | awk '{print $1}' | sort | uniq -c
  • ESTABLISHED: Active connections currently exchanging data.
  • TIME_WAIT: Connections closed by Nginx but the OS holds the socket for a few minutes before cleaning it up (to make sure remaining network data packets on the way arrive). If the TIME_WAIT count is too high (tens of thousands), this is a sign that we’re not turning on HTTP Keepalive on our server, so every request triggers opening and closing a new TCP socket.
  • CLOSE_WAIT: Indicates our backend application server is slow to close sockets after the connection finishes. Too many CLOSE_WAIT is an indication of a connection leak on the backend application side.

strace: Low-Level Syscall Debugging #

When Nginx returns a strange error in the log like Permission denied but we’ve already fixed the file chown/chmod, we can use strace to see what happens at the Linux kernel level when the Nginx worker process tries to access the file.

strace monitors and records the system calls made by a running process.

# 1. Find the PID number of one of the active Nginx worker processes
pgrep -x nginx

# 2. Trace syscalls specifically for file and network operations on that PID
sudo strace -p <WORKER_PID> -e trace=file,network

When we send a request from the browser, strace immediately dumps syscall lines in the terminal in real-time:

openat(AT_FDCWD, "/var/www/html/secret.txt", O_RDONLY) = -1 EACCES (Permission denied)

From the output above, it’s clear the openat syscall failed with the EACCES code because it’s blocked by a security policy (like AppArmor or SELinux), proving the problem isn’t in Nginx syntax but the operating system security restriction.

tcpdump: Low-Level Network Traffic Analysis #

When Nginx returns a mysteriously disconnected connection error (e.g., clients failing to send large request bodies or upstream connections cutting off mid-way), and strace doesn’t provide enough information, we must go down to the network packet level using tcpdump.

tcpdump records TCP/IP packets crossing our server’s network interface.

  • Capturing TCP Packets on a Certain Port (Port 80/443):

    # Capture packets on the eth0 interface going to or coming from port 443
sudo tcpdump -i eth0 port 443 -vv -XX
# -i: interface, -vv: very verbose, -XX: show data in hexadecimal and ASCII format

  • Saving Capture Results for Wireshark: Reading packet lines in the terminal is very complicated. The best practice is recording packets into a .pcap file then analyzing them using the Wireshark visual application on our local computer:

    # Capture the first 1000 packets on port 443 and save to a dump file
sudo tcpdump -i any port 443 -c 1000 -w /tmp/ssl_handshake.pcap

  • Detecting TCP Resets (RST): In connection reset debugging cases, we can filter tcpdump packets to only look for the TCP RST flags sent by Nginx or the backend:

    sudo tcpdump -i any 'tcp[tcpflags] & tcp-rst != 0' -nn

    If we see a flow of RST packets right after the TLS Client Hello is sent, there’s a TLS cipher suite mismatch at the network level.

Diagnosing File Descriptor (FD) Leaks #

Nginx consumes one File Descriptor (FD) for every static file opened, log file written, and every active TCP socket connection (client and upstream). If our Nginx server experiences high traffic and suddenly refuses connections with the log message socket() failed (24: Too many open files), Nginx has reached the maximum file descriptor limit allowed by the Linux kernel.

We can diagnose this obstacle using the following commands:

  1. Check the Active Nginx Process Limit: We can see the soft and hard file descriptor limit values currently in effect on our Nginx master process:

    # Get the Nginx master PID
PID=$(cat /run/nginx.pid)

# Display the file descriptor limits (Max open files)
cat /proc/$PID/limits | grep "Max open files"
# Standard output is usually: Soft Limit 1024, Hard Limit 4096

  2. Count the FDs Currently in Use: We can count how many FDs are currently actively opened by all Nginx worker processes:

    # Count the number of open file descriptors in /proc
sudo lsof -c nginx | wc -l

  3. Track the Types of Open FDs: Use the lsof command to see details of what files or socket connections Nginx is currently holding:

    sudo lsof -p $PID
# The output shows the FD types: REG (regular files), IPv4/IPv6 (TCP sockets), or FIFO (pipes)

    If we see thousands of sockets in the can't identify protocol status, that indicates the backend upstream is hanging and socket connections are being held dangling in memory (descriptor leak).

ab and wrk: Stability Load Testing #

Before publishing new performance optimization configurations to production environments, we must test the server’s endurance limits using load testing tools.

1. Using Apache Benchmark (ab) #

ab is very practical for quick concurrency testing:

# Send a total of 5000 requests with 50 simultaneous persistent connections (keepalive)
ab -n 5000 -c 50 -k https://app.unisbadri.com/

2. Using wrk (Multi-Threaded) #

wrk is much more modern and can simulate massive traffic loads because it runs multi-threaded utilizing all CPU cores of our laptop:

# Run a load test for 30 seconds using 4 threads and 100 simultaneous connections
wrk -t4 -c100 -d30s https://app.unisbadri.com/

The wrk output shows the average latency metric (Latency) and throughput of requests per second (Requests/sec). If we see a high error percentage, the file descriptor limit or upstream buffer configuration in our Nginx needs adjusting.

GoAccess: Visual Real-Time Log Analysis #

Reading Nginx log text files manually using tail or grep commands is helpful for one case, but hard to use for seeing overall traffic trend patterns.

GoAccess is an interactive log analyzer that directly reads Nginx access log files and displays them as a beautiful visual dashboard in the terminal or exports them as a dynamic HTML page.

# Run interactive real-time log analysis in the terminal
sudo goaccess /var/log/nginx/access.log --log-format=COMBINED

# Generate a visual dashboard in HTML file format in one go
sudo goaccess /var/log/nginx/access.log --log-format=COMBINED -o /var/www/html/report.html

The GoAccess HTML dashboard automatically updates its visualizations every time a new log line comes in. We can monitor the most active visitor IPs, the API routes that are most often slow, up to the status code distribution (2xx, 3xx, 4xx, 5xx) instantly.

Prometheus + Grafana: Continuous Monitoring #

To monitor Nginx server health long-term, we can’t rely on manual commands. We must set up automatic monitoring metrics using Prometheus and visualize them in Grafana.

1. Enable stub_status in Nginx #

Prometheus needs a raw metrics endpoint to pull from. We enable Nginx’s built-in status module:

server {
    listen 127.0.0.1:8080;
    server_name localhost;

    location /nginx_status {
        stub_status on;
        access_log off;
        allow 127.0.0.1;
        deny all;
    }
}

2. Run the nginx-prometheus-exporter #

The exporter reads the local Nginx /nginx_status page and translates it into the metrics format understood by Prometheus:

# Run the exporter using a Docker container
docker run -d \
    -p 9113:9113 \
    nginx/nginx-prometheus-exporter:latest \
    -nginx.scrape-uri=http://localhost:8080/nginx_status

3. Configure the Scrape Target in Prometheus #

Add the exporter target to the prometheus.yml configuration file:

scrape_configs:
  - job_name: 'nginx_exporter'
    static_configs:
      - targets: ['localhost:9113']

Through Grafana, we can map historical graphs of crucial metrics like:

  • nginx_connections_active: The number of active connections currently being handled by Nginx.
  • nginx_connections_waiting: Idle keepalive connections waiting for the next request.
  • nginx_http_requests_total: The total accumulated requests coming into our server.

Quick Diagnostic Checklist #

Here are the emergency steps we must execute when the Nginx server experiences obstacles:

# 1. Make sure the Nginx process is actively running
sudo systemctl status nginx

# 2. Verify the configuration syntax cleanliness
sudo nginx -t

# 3. Check whether ports 80 and 443 are listening
sudo ss -tlnp | grep nginx

# 4. Show the latest 50 error lines to look for emerg/error messages
sudo tail -n 50 /var/log/nginx/error.log

# 5. Test the backend response directly without going through Nginx
curl -I http://localhost:3000/health

# 6. Make sure requests really reach Nginx
sudo tail -f /var/log/nginx/access.log

# 7. Check whether DNS resolution points to the correct server IP
dig +short app.unisbadri.com

# 8. Make sure the firewall isn't blocking external port access
sudo ufw status || sudo iptables -L -n

Summary #

  • curl -v: The mandatory first step for checking the HTTP conversation flow, TLS handshake, and response header data.
  • openssl s_client: Use specifically for solving SSL errors, validating certificate expiration dates, and testing TLS protocol compatibility.
  • ss -tn: Monitor the TCP state distribution to detect idle socket buildup (TIME_WAIT) caused by the absence of keepalive.
  • strace: Trace system calls at the OS kernel level to diagnose hidden file permission obstacles.
  • GoAccess: A fast real-time visual analysis solution without needing to build a complex ELK Stack infrastructure.

← Previous: Config Debugging   Next: Best Practices →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact