Error Log #

If the access log acts as a data traffic diary recording every normal activity, then the Error Log is our server’s medical diagnosis record. Unlike the access log, which records all interactions without exception, the error log is designed exclusively to document failures, security warnings, configuration mismatches, and other abnormal conditions Nginx encounters while operating.

Being able to read, interpret, and trace messages in the error log is an absolute skill system administrators must master. Through a deep understanding of the error log, we can detect backend application failures before users feel them, identify exploit scanning attacks, and fix internal network performance problems. In this article, we’ll discuss the 8 error log severity levels, dissect the anatomy of error messages line by line, learn how to solve common production problems, and use per-IP debug techniques safely in production environments.

The Fundamental Difference: Access Log vs Error Log #

Before going deeper, let’s clarify the fundamental difference between Nginx’s two main log files so we don’t look for the wrong information when troubleshooting:

Understanding the 8 Severity Levels #

Nginx classifies every event in the error log into one of eight severity levels. These levels are derived from the Syslog logging standard.

Here’s the severity order from the most critical to the most detailed (verbose):

flowchart TD
    emerg("1. emerg - Critical - System Unstable")
    alert("2. alert - Needs Immediate Action")
    crit("3. crit - Critical Condition")
    error("4. error - Request/System Error")
    warn("5. warn - Warning/Temporarily Ignorable")
    notice("6. notice - Significant Information")
    info("7. info - General Information")
    debug("8. debug - Very Verbose - Development")

    emerg --> alert
    alert --> crit
    crit --> error
    error --> warn
    warn --> notice
    notice --> info
    info --> debug

    classDef default fill:#f9f9f9,stroke:#d1d5db,stroke-width:1px,color:#111827;
    classDef dangerStyle fill:#fee2e2,stroke:#ef4444,stroke-width:2px,color:#991b1b;
    classDef warnStyle fill:#fef3c7,stroke:#f59e0b,stroke-width:2px,color:#92400e;
    classDef infoStyle fill:#eff6ff,stroke:#3b82f6,stroke-width:2px,color:#1e3a8a;
    
    class emerg,alert,crit dangerStyle;
    class error,warn warnStyle;
    class notice,info,debug infoStyle;

Let’s break down the characteristics of each level:

1. emerg (Emergency): An emergency condition where the Nginx server can’t run or is in a very unstable state (e.g., completely running out of system memory allocation or a critical failure in main OS libraries).
2. alert: A serious problem requiring immediate human intervention (e.g., failure to open an important socket file, an internal database failure locking the master process, or reaching the OS file descriptor limit).
3. crit (Critical): A critical event causing some Nginx functions to be hampered (e.g., memory allocation failure for a new worker process or failure to open a configuration file during reload).
4. error: A standard operational problem causing a client request to fail processing (e.g., failure to connect to an upstream backend, a local file not found, or an SSL handshake problem with a client).
5. warn (Warning): A warning about suspicious behavior or deprecated configuration usage, but not stopping the response delivery process (e.g., log buffer size too small or overlapping duplicate host configurations).
6. notice: Operational events running normally but with important significance for administrators (e.g., the Nginx master process detecting configuration changes after reload, or a new worker process successfully starting).
7. info: General information messages useful for monitoring system activity (e.g., SSL connection renegotiation information).
8. debug: Very detailed low-level internal tracing information about every socket operation, buffer read, and C code flow. This level is very verbose.

The Effect of the Logging Threshold Level #

When we set a certain severity level in the configuration, Nginx records events at that level along with all levels above it.

For example, if we set the level to warn, Nginx records events categorized as warn, error, crit, alert, and emerg. Messages categorized as notice, info, and debug are ignored.

# Global level configuration (usually at the top of nginx.conf)
# Highly recommended to set the level to 'warn' for production environments
error_log /var/log/nginx/error.log warn;

[!WARNING] Never set the error log level to debug in a production environment. The debug level writes millions of lines of information every minute, which will free up CPU performance only to write logs, fill up disk space within hours, and drastically slow down our web application’s response time.

Anatomy of an Nginx Error Message #

Nginx error messages are designed with a dense, information-rich structured format to help trace root causes. Let’s dissect an example production error message line:

2026/06/16 13:20:45 [error] 4821#4821: *30412 connect() failed (111: Connection refused) while connecting to upstream, client: 203.0.113.102, server: api.unisbadri.com, request: "POST /v1/users HTTP/2.0", upstream: "http://127.0.0.1:8080/v1/users", host: "api.unisbadri.com"

If we dissect that log line, we get the following information elements:

1. 2026/06/16 13:20:45: The event timestamp in server local format (Year/Month/Day Hour:Minute:Second).
2. [error]: The event severity level.
3. 4821#4821: The PID (Process Identifier) of the Nginx worker process handling that connection, followed by the internal thread ID.
4. *30412: The unique internal Nginx request ID number (connection ID). This is very useful because if a request triggers several error messages, we can trace them by searching for the same *30412 ID across the whole log file.
5. connect() failed (111: Connection refused) while connecting to upstream: The detailed error description. Here Nginx informs us that the connect() system call to the backend failed with OS error code 111 (Connection refused) while trying to forward the request to an upstream.
6. client: 203.0.113.102: The real client IP address that triggered this request.
7. server: api.unisbadri.com: The server block (virtual host) domain name where the error occurred.
8. request: "POST /v1/users HTTP/2.0": The original HTTP request line, showing the method, path, and protocol used by the client.
9. upstream: "http://127.0.0.1:8080/v1/users": The upstream/backend destination address that failed to connect (in this case, the local backend application running on port 8080).
10. host: "api.unisbadri.com": The HTTP Host header value sent by the client.

Common Problems and How to Solve Them #

Here’s a compilation of the most frequently appearing error messages on production Nginx servers, the meaning behind those messages, common causes, and solution steps to fix them:

1. connect() failed (111: Connection refused) while connecting to upstream #

• Meaning: Nginx successfully received the request from the client and tried to forward it to a backend server (like PHP-FPM, Node.js, Go, Python), but the backend server refused the connection.
• Common Causes:
  • Our backend application isn’t running (crashed or never started).
  • The backend is listening on a different port or Unix socket path than in the Nginx configuration.
  • A local firewall (like UFW or iptables) blocks that internal port communication.
• Solution:
  • Check our backend’s status (e.g., systemctl status php8.2-fpm or pm2 status).
  • Make sure the port in the proxy_pass http://127.0.0.1:PORT; directive matches our backend application’s configuration.

2. no live upstreams while connecting to upstream #

• Meaning: Nginx marked all backend servers declared in the upstream block as inactive (down) due to previous interaction failures (passive healthcheck).
• Common Causes:
  • All backend instances crashed simultaneously.
  • The max_fails and fail_timeout parameters on the upstream configuration are set too tight, so small backend fluctuations immediately make Nginx block traffic delivery to it.
• Solution:
  • Fix the dead backend.
  • Loosen the health parameters in our upstream block:

    upstream backend_servers {
        server 10.0.0.10:3000 max_fails=3 fail_timeout=30s;
        server 10.0.0.11:3000 max_fails=3 fail_timeout=30s;
    }
    

3. upstream timed out (110: Connection timed out) while reading response header from upstream #

• Meaning: Nginx successfully connected to the backend, but the backend took too long to process data and didn’t send a response back until the time limit was exceeded.
• Common Causes:
  • Database queries in our backend application are very slow.
  • The backend has a memory leak or is under CPU overload.
  • The Nginx timeout value (proxy_read_timeout) is set too low for legitimate transactions (like monthly report processing).
• Solution:
  • Optimize our backend application code.
  • Raise the response time limit in our Nginx server configuration block:

    location /api/laporan {
        proxy_read_timeout 300s;
        proxy_pass http://backend_servers;
    }
    

4. (13: Permission denied) while reading response header from upstream #

• Meaning: Nginx is blocked by the operating system’s security system while trying to read or write data.
• Common Causes:
  • On RedHat/CentOS/Rocky Linux systems, the SELinux security feature by default blocks Nginx from making outbound network connections to backend ports.
  • The Unix socket file (e.g., /var/run/php-fpm.sock) has file permissions that can’t be read by the Nginx system user (www-data or nginx).
• Solution:
  • If caused by SELinux, run the command:

    sudo setsebool -P httpd_can_network_connect 1
    

  • If caused by socket file permissions, adjust the socket owner in the PHP-FPM/backend configuration so Nginx has read/write access rights.

5. open() "/var/www/html/missing.html" failed (2: No such file or directory) #

• Meaning: The client requested a static file that doesn’t exist in our server’s physical directory.
• Common Causes:
  • The client mistyped the URL (causing HTTP 404).
  • The root or alias directive in the Nginx configuration points to the wrong folder on the server.
• Solution:
  • Make sure the path in the root directive points to the correct folder.
  • If our application is a Single Page Application (SPA) like React/Vue/Angular, make sure we use the try_files directive so requests are redirected to index.html and handled by frontend routing:

    location / {
        root /var/www/my-spa;
        try_files $uri $uri/ /index.html;
    }
    

Connection Debug Technique: Safe Debug Logging in Production #

When there’s an error report from a specific user in the production environment, we’re often tempted to raise the global log level to debug to monitor request details in depth. However, as discussed earlier, this action is very dangerous for server performance and disk storage space.

To solve this problem, Nginx provides a clever solution called the debug_connection directive. This directive is placed inside the events context block and lets us enable debug logging only for a specific IP address or CIDR segment.

Here’s an example configuration:

# Put this configuration in the /etc/nginx/nginx.conf file
error_log /var/log/nginx/error.log warn; # The global log level stays safely at 'warn'

events {
    worker_connections 1024;

    # Enable the debug level ONLY for the IPs below
    # (E.g., our internal developer team's IPs or a specific user's IP experiencing errors)
    debug_connection 203.0.113.205;
    
    # We can also specify an entire CIDR subnet block
    debug_connection 10.10.0.0/24;
}

How debug_connection Works #

When a request comes from IP 203.0.113.205, Nginx switches that request’s logging processing to the debug level, recording the entire detailed internal transaction history to the /var/log/nginx/error.log file. Meanwhile, for millions of traffic from other users coming from different IPs, Nginx keeps applying the memory- and storage-efficient warn logging rule.

This is a very valuable advanced technique for safely diagnosing strange SSL handshake problems or request parameter anomalies at the production level.

Managing Error Logs per Virtual Host #

Just like the access log, we should always isolate error log files for each domain so the triage and problem identification process isn’t hampered by other domains’ data being mixed in.

# Virtual Host: portal.unisbadri.com
server {
    listen 80;
    server_name portal.unisbadri.com;

    # Isolate the error log with a specific level for the portal
    access_log /var/log/nginx/portal-access.log combined;
    error_log /var/log/nginx/portal-error.log warn;

    location / {
        root /var/www/portal;
        index index.html;
    }
}

By isolating error logs per server block, if our monitoring team gets an alert that the portal domain is experiencing problems, they can simply focus on analyzing the /var/log/nginx/portal-error.log file.

Quick Error Log Analysis via CLI #

Here are some practical CLI commands to speed up problem tracing in our Nginx error log file:

1. Monitoring New Errors in Real-time (Very useful when deploying a new application) #

sudo tail -f /var/log/nginx/error.log

2. Searching for a Specific Error Type (E.g., backend upstream connection problems) #

sudo grep -i "connect() failed" /var/log/nginx/error.log

3. Counting Event Occurrences by Severity Level (Seeing the error severity distribution) #

sudo grep -oP '\[\K[^\]]+' /var/log/nginx/error.log | sort | uniq -c | sort -rn

• How it works: This command uses a regular expression to extract strings inside square brackets [...] (which contain severity levels like error, warn, crit), then counts their occurrence frequency.

4. Finding the Client IPs That Most Frequently Trigger Errors #

sudo grep -oP 'client: \K[^,]+' /var/log/nginx/error.log | sort | uniq -c | sort -rn | head -n 10

If we see one IP triggering thousands of 404 or Permission denied errors in a short time, that’s a strong indicator the IP is doing automated scanning activity (vulnerability scanning) on our site. We can immediately block that IP using a firewall or the IP restriction module.

Summary and Best Practices #

• Set the Severity Level to warn: In production environments, always set the error_log directive to the warn severity level to balance troubleshooting information needs with server performance efficiency.
• Use debug_connection: Use debug connections to selectively enable debug level logging per client IP without affecting overall server performance.
• Separate Error Logs per Virtual Host: Declare different error log files on each domain server block to keep our server administration clean.
• Analyze Logs Periodically: Use standard CLI commands like grep, awk, and uniq to monitor error trends and identify traffic anomalies that could become security threats.

← Previous: Access Log   Next: Custom Log Formats →