Log Rotation #

The Nginx log files we enable will keep growing over time and with the volume of traffic coming into our server. On production servers with medium to heavy traffic, access log files can easily accumulate gigabytes of data within weeks or even days. If left unmanaged, these ever-growing log files will consume our server’s entire disk storage capacity, causing overall system failure and slowing down log reader utility performance.

To overcome this challenge, we implement Log Rotation. Log rotation is an automation process for periodically cutting off the active log file, compressing old log files into archive format to save storage space, limiting the number of stored archive files (retention), and automatically creating clean new log files. In this article, we’ll learn the internal mechanism of how Nginx holds log files, how the USR1 signal is used to reopen log files, dissect the standard logrotate configuration file in Linux, and do our own log storage testing and monitoring.

Why Do We Need Log Rotation? #

Piling all traffic data into one single huge file (e.g., a 50 GB access.log file) creates several serious operational problems for us:

  1. Disk Exhaustion Risk: This is the most real danger. If the disk partition where logs are stored fills to 100%, the Linux operating system can’t write new data. Database services stop, caching systems freeze, and Nginx itself may fail to process new requests.
  2. Utility Performance Degradation: Trying to read, search for keywords (using grep or awk), or analyze a tens-of-gigabytes log file takes a very long time and consumes large amounts of server RAM.
  3. Legal Compliance: Data protection regulations often limit how long user activity logs can be stored (e.g., only 30 days or 90 days). With log rotation, we can automatically delete old log files that have passed the retention limit to comply with legal rules.

How Nginx Handles Log Files #

There’s one crucial technical concept about the Linux operating system and Nginx that every system administrator must understand: Nginx holds an open file descriptor to the log file continuously.

When an Nginx worker process writes a log line to /var/log/nginx/access.log, it doesn’t look up the file by name every time it writes. Nginx uses the file descriptor number obtained when it first opened that file.

This triggers the following behavior:

  • If we rename the access.log file to access.log.1, Nginx will still write logs to that access.log.1 file. Nginx won’t automatically create a new access.log file because the file descriptor it holds still points to the same file inode on disk.
  • Even if we delete the access.log file directly with the rm command, Nginx will still write to that file descriptor. The disk space won’t be freed by the operating system until we tell Nginx to close the old file descriptor.

The USR1 Signal (nginx -s reopen) #

To tell Nginx to release the old file descriptor and open a clean new log file, we must send a system signal named USR1 to the Nginx master process.

After receiving the USR1 signal, the Nginx master process orders all worker processes to:

  1. Close the currently open log file.
  2. Reopen the log file using the original file name written in the Nginx configuration (which will detect that the file is empty or newly created).
  3. Create a new log file with appropriate access permissions if the file doesn’t exist on disk yet.

Here’s the sequence diagram flow of the log rotation process in Nginx:

sequenceDiagram
    autonumber
    participant LR as Logrotate Agent
    participant OS as OS File System
    participant NX as Nginx Master Process

    LR->>OS: Rename access.log to access.log.1
    Note over NX: Nginx keeps writing logs to<br/>access.log.1 via the file descriptor
    LR->>OS: Create a new empty access.log file (optional)
    LR->>NX: Send the USR1 signal (nginx -s reopen)
    NX->>OS: Close the access.log.1 file descriptor
    NX->>OS: Open a new file descriptor to access.log
    Note over NX: Nginx starts writing logs to<br/>the new access.log file
    LR->>OS: Compress access.log.1 into access.log.1.gz

The manual command to send this USR1 signal is:

# Method 1: Use the built-in nginx command
sudo nginx -s reopen

# Method 2: Send a direct kill signal to the Nginx master PID
sudo kill -USR1 $(cat /var/run/nginx.pid)

Standard logrotate Utility Configuration #

On most Linux distributions (like Ubuntu, Debian, Rocky Linux, CentOS), we don’t need to write our own log rotation script. The operating system includes a standard utility called logrotate that runs periodically (usually as a daily cron job or systemd timer).

When we install Nginx through a package manager (apt or dnf), the installer automatically places the Nginx logrotate configuration file at /etc/logrotate.d/nginx.

Let’s dissect the standard /etc/logrotate.d/nginx configuration content to understand the functionality of each parameter:

/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 `cat /var/run/nginx.pid`
        fi
    endscript
}

Parameter Explanation Line by Line #

  • /var/log/nginx/*.log: Specifies the target log files to be processed by the rules inside the curly braces. In this case, all files ending in .log inside the /var/log/nginx/ directory.
  • daily: Specifies the rotation frequency. Log files are rotated every day. Other options are weekly, monthly, or yearly.
  • missingok: If for some reason the log file isn’t found (e.g., we deleted it manually), logrotate ignores it and continues the process without triggering a system error message.
  • rotate 14: Specifies log retention. We only keep a maximum of 14 rotated old log files. On day 15, the oldest log file (access.log.14.gz) is permanently deleted to save disk space.
  • compress: Compresses rotated old log files using gzip to save up to 80-90% of storage space.
  • delaycompress: Delays the compression process by one rotation cycle. The log file just rotated today (access.log.1) won’t be immediately compressed to .gz. It will only be compressed on the next day’s rotation (becoming access.log.2.gz). This is very important because it gives Nginx worker processes time to actually finish writing and close their old file descriptors after receiving the reopen signal.
  • notifempty: Don’t rotate if the active log file is empty (no data traffic). This saves disk space and prevents creating useless empty archive files.
  • create 0640 www-data adm: After renaming the old log file, logrotate immediately creates a new empty log file with 0640 access permissions (owner can read/write, group can read, others can’t access at all), with www-data user and adm group ownership.
  • sharedscripts: Instructs logrotate to run the script inside the postrotate block only once after all log files have been processed, not once for each log file found. This prevents the Nginx server from being reloaded repeatedly if many virtual host log files are rotated at the same time.
  • postrotate ... endscript: The shell command block executed after the rotation process finishes. This is where we put the kill -USR1 command to tell Nginx to reopen the new empty log files.

Customizing Log Rotation Rules for Specific Virtual Hosts #

The default Nginx logrotate configuration treats all log files uniformly. However, in production environments, we might want to keep logs from certain virtual hosts longer (for business transaction compliance reasons) compared to static asset logs or less important test subdomain logs.

We can set this up by separating the log file target blocks in the /etc/logrotate.d/nginx configuration:

# Rule 1: Main API Log (Kept 90 days for compliance audits)
/var/log/nginx/api.unisbadri.com-*.log {
    daily
    rotate 90
    compress
    delaycompress
    missingok
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 `cat /var/run/nginx.pid`
        fi
    endscript
}

# Rule 2: Static Asset Log (3 days is enough since it only contains CSS/JS/Image file requests)
/var/log/nginx/static-assets-*.log {
    daily
    rotate 3
    compress
    delaycompress
    missingok
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 `cat /var/run/nginx.pid`
        fi
    endscript
}

With this configuration, we intelligently optimize our server’s disk storage capacity by not keeping low-value log data for too long.

Testing and Executing Log Rotation Manually #

After changing the Nginx logrotate configuration, we must test it to make sure there are no syntax errors that could break the daily automatic rotation process.

1. Dry Run Simulation #

We can run a log rotation simulation without actually changing any files on disk using the --debug parameter:

sudo logrotate --debug /etc/logrotate.d/nginx

This command displays a step-by-step log of what logrotate would do, which files would be renamed, and which postrotate commands would be run.

2. Force Rotation Now #

If our disk capacity is almost full and we want to trigger log rotation immediately without waiting for the system’s automatic schedule:

sudo logrotate --force /etc/logrotate.d/nginx

This command forces logrotate to rotate all Nginx log files, create new files, compress old files, and reload Nginx.

3. Checking the Last Rotation Status #

The logrotate utility stores the last execution date log of every configuration rule in a status file:

sudo cat /var/lib/logrotate/status | grep nginx

Log Rotation for Nginx in Docker Containers #

When we run Nginx inside a Docker container, the traditional log rotation approach using logrotate at the host OS level can’t be applied directly. By default, the official Docker Nginx image configures the access log and error log to be redirected to /dev/stdout (standard output) and /dev/stderr (standard error):

# Default configuration inside the Docker Nginx image
access_log /dev/stdout;
error_log /dev/stderr;

The purpose of this redirection is so all Nginx logs can be captured directly by the Docker Daemon and consumed via the docker logs <container_id> command.

In this Docker architecture, log rotation isn’t done inside Nginx, but is delegated to the Docker daemon level using a logging driver. If we don’t limit it, the Docker JSON log files on the host server (/var/lib/docker/containers/*/*-json.log) will grow without bound.

Docker Daemon Log Rotation Configuration #

We must configure the Docker log size limits globally in the /etc/docker/daemon.json file on our host server:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}
  • max-size: "10m": Docker rotates the container log when the log file size reaches 10 Megabytes.
  • max-file: "3": Docker only keeps a maximum of 3 old log files per container.

After configuring the file, we must restart the Docker service:

sudo systemctl restart docker

If we still want to write logs to local files inside the Docker container (e.g., because we need a separate JSON log file for Promtail/Filebeat), we must mount that log directory to the host server using a Docker volume, then configure logrotate at the host server level to rotate those files and send the USR1 signal to the container using the command:

# Special postrotate command for Docker Nginx containers
docker exec -it <container_name_or_id> nginx -s reopen

Manual Log Rotation Using a Lightweight Cron Script #

On minimalist systems without the logrotate utility installed (like some minimal Alpine Docker installs or IoT servers), we can write our own simple shell script to do log rotation safely, then schedule it using the built-in cron.

Here’s an example of a safe shell script (/usr/local/bin/rotate-nginx.sh):

#!/bin/sh

# Define path configuration
LOG_DIR="/var/log/nginx"
PID_FILE="/var/run/nginx.pid"
BACKUP_DATE=$(date +"%Y%m%d_%H%M%S")

# 1. Go into the log directory
cd "$LOG_DIR" || exit 1

# 2. Rename the current active files
for log_file in *.log; do
    # Skip if there's no active log file
    [ -f "$log_file" ] || continue
    
    # Rename the active file to a timestamped backup file
    mv "$log_file" "${log_file}.${BACKUP_DATE}"
done

# 3. Send the USR1 signal to Nginx to create clean new log files
if [ -f "$PID_FILE" ]; then
    kill -USR1 "$(cat "$PID_FILE")"
else
    echo "Warning: Nginx PID not found, log reopen failed."
fi

# 4. Wait 2 seconds for Nginx worker processes to close old FDs
sleep 2

# 5. Compress the newly created backup log files
for backup_file in *.log.[0-9]*; do
    [ -f "$backup_file" ] || continue
    gzip "$backup_file"
done

# 6. Delete backup logs older than 14 days
find "$LOG_DIR" -name "*.log.*.gz" -mtime +14 -exec rm -f {} \;

After creating the script, don’t forget to give it execution permissions:

sudo chmod +x /usr/local/bin/rotate-nginx.sh

Then, we register the script in the root crontab (sudo crontab -e) to run every night at 23:59:

59 23 * * * /usr/local/bin/rotate-nginx.sh >/dev/null 2>&1

Monitoring Log Disk Storage Capacity #

As part of routine maintenance, we should always monitor the disk space consumption used by our Nginx log files.

1. Checking Log File Sizes #

We can check which log files are consuming the most storage capacity using the du (disk usage) command:

# Show the size of each log file in the Nginx directory
sudo du -sh /var/log/nginx/* | sort -h

2. Finding Oversized Log Files Exceeding the Limit #

If we want to scan whether any log file is growing too fast, exceeding 1 GB:

sudo find /var/log/nginx/ -type f -name "*.log" -size +1G

3. Safely Deleting Old Logs #

Never delete an active log file with the rm command without reloading Nginx. The safest way to instantly clear an active log file’s contents without disturbing Nginx is to truncate it:

# Safely empty the active log file to 0 bytes
sudo truncate -s 0 /var/log/nginx/access.log

The truncate command removes all data content inside the file but keeps the file descriptor and inode, so Nginx can immediately continue writing without errors.

Summary and Best Practices #

  • Always Use delaycompress: Make sure the delaycompress parameter is always enabled together with compress so Nginx has enough time to close old file descriptors before gzip compression happens.
  • Use sharedscripts: Use the sharedscripts parameter so the USR1 signal script call to Nginx is only executed once after all log files have been processed.
  • Empty Logs Safely: If you must clean an active log urgently, use the truncate -s 0 file.log command instead of rm or echo "" > file.log.
  • Separate Log Retention: Differentiate the retention duration of sensitive API logs (keep longer) from static asset logs (keep shorter) to save disk space.
  • Do a Dry Run After Editing: Always run logrotate --debug /etc/logrotate.d/nginx after changing rotation rules to catch configuration errors early.

← Previous: Custom Log Formats   Next: Worker Process →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact