Open File Cache #

Every time Nginx serves a static file (like HTML files, images, CSS documents, JavaScript scripts, or fonts) directly from the server’s local storage to the client, the operating system in the background must perform a series of system calls (syscalls). These system calls include the process of finding the file’s physical location on the storage system (inode lookup), reading the file metadata (size and last modification time), opening the file descriptor, transmitting the data, and finally closing the file descriptor again.

On production servers serving thousands of static file requests per second, the repeated execution of syscalls like open(), stat(), and close() can become a serious performance bottleneck (I/O bottlenecks) for our server CPU. Nginx solves this problem by providing the Open File Cache feature. This feature stores open file descriptors along with their metadata directly in Nginx’s RAM. In this article, we’ll discuss how Open File Cache works, dissect its configuration parameters, put together the synergistic “static performance trio” configuration, and analyze the scenarios where this feature provides the most significant impact.

The System Call Overhead Problem #

To understand the importance of the Open File Cache, let’s examine the Linux operating system’s workflow when Nginx serves one static file without descriptor caching:

open() Syscall: Nginx asks the Linux kernel to open the file (e.g., /var/www/static/js/app.js). The kernel must traverse the directory structure on disk to find the file’s inode number.
fstat() Syscall: Nginx asks the kernel to read the file metadata to know the file size (for the Content-Length header) and last modification time (for the Last-Modified header).
Data Transmission: Nginx reads the file into memory and sends it to the network socket.
close() Syscall: Nginx closes the file descriptor again to release operating system resources.

Each of the system calls above requires a CPU context switch from user space (the Nginx application area) to kernel space (the operating system core area) and back again. If our server must serve 10,000 static file requests per second, our server CPU wastes most of its computing power just going back and forth doing context switches and disk inode lookups. This problem feels much more severe if our storage system has high lookup latency (like on Network File Systems - NFS, or regular mechanical hard disks).

How `open_file_cache` Works in Nginx #

By enabling the open_file_cache directive, Nginx stores open file descriptor data along with its metadata information in Nginx’s RAM.

Here’s a comparison flow diagram of static file lookup without vs with the open file cache:

flowchart TD
    Request["Client Requests a Static File"] --> CheckCache{"Is the File Descriptor<br/>in the RAM Cache?"}
    
    CheckCache -->|Yes: Cache HIT| SendFile["Directly Send the File via sendfile"]
    CheckCache -->|No: Cache MISS| SyscallOpen["Call the open & stat Syscalls to the OS"]
    
    SyscallOpen --> DiskLookup["OS Searches the Inode on Disk Storage"]
    DiskLookup --> OpenFD["Open the File Descriptor (FD) & Read Metadata"]
    OpenFD --> SaveRAM["Save the FD & Metadata to the RAM Cache"]
    SaveRAM --> SendFile

    classDef default fill:#f9f9f9,stroke:#d1d5db,stroke-width:1px,color:#111827;
    classDef hitStyle fill:#d1fae5,stroke:#10b981,stroke-width:2px,color:#065f46;
    classDef missStyle fill:#fee2e2,stroke:#ef4444,stroke-width:2px,color:#991b1b;
    classDef processStyle fill:#eff6ff,stroke:#3b82f6,stroke-width:2px,color:#1e3a8a;
    
    class SendFile hitStyle;
    class CheckCache processStyle;
    class SyscallOpen,DiskLookup,OpenFD missStyle;

When the cache is active and the next request for the same file arrives:

Nginx detects that the file descriptor is already registered in its RAM cache.
Nginx skips the open() and fstat() system calls to the operating system.
Nginx directly sends the file data using the already-open file descriptor.
The file descriptor is left open in memory for subsequent requests.

This dramatically cuts CPU workload, eliminates disk filesystem lookup latency, and speeds up response delivery time to users.

Configuring the `open_file_cache` Directive #

The open_file_cache directive can be declared in the global http context, a server block, or at a specific location level.

Basic Configuration Syntax #

http {
    # max: the maximum number of entries stored in the cache
    # inactive: remove files from the cache if not accessed within this time
    open_file_cache max=10000 inactive=30s;
}

max=10000: Nginx holds a maximum of 10,000 active file descriptor entries in its RAM. If the number of unique files accessed exceeds 10,000, Nginx automatically evicts the least-frequently-accessed entries (Least Recently Used - LRU) to make room for new files. The memory consumed is very small, only about 500 bytes per entry, so 10,000 entries only take about 5 Megabytes of RAM.
inactive=30s: Determines the inactivity retention time limit. If a cached file isn’t accessed by any client for 30 seconds, the entry is immediately removed from RAM to ensure RAM isn’t filled with one-hit wonder files.

Open File Cache Fine-Tuning Configuration #

To control how Nginx validates and maintains data accuracy inside the cache, we use three additional directives:

http {
    open_file_cache max=10000 inactive=30s;

    # 1. How often Nginx validates cache data correctness against the disk
    open_file_cache_valid 60s;

    # 2. Minimum access count before a file is put into the cache
    open_file_cache_min_uses 2;

    # 3. Enable caching for error messages (errors)
    open_file_cache_errors on;
}

Understanding the Alignment Directives’ Roles #

open_file_cache_valid 60s;: When file descriptors are stored in memory, there’s a risk the file is changed or updated by our developers on disk. Nginx won’t detect that change instantly because it reads descriptor data from memory. This directive instructs Nginx to check the disk every 60 seconds to verify whether the file has been modified or deleted. Setting this value higher (e.g., 5 minutes) improves performance but slows down static file update detection in users’ browsers.
open_file_cache_min_uses 2;: To prevent random rarely-accessed files (e.g., old PDF download documents) from filling the max cache quota, Nginx only puts a file descriptor into the cache if the file has been accessed at least 2 times within the time range determined by the inactive parameter (30 seconds).
open_file_cache_errors on;: Instructs Nginx to also cache error messages like 404 Not Found (file doesn’t exist) or 403 Forbidden (permission denied). If an attacker bot tries scanning thousands of random fake URLs to our server, Nginx doesn’t need to do repeated disk filesystem scans for every spam request—it can just answer instantly from its RAM error cache.

The Static Performance Trio: `open_file_cache` + `sendfile` + `tcp_nopush` #

Nginx’s static performance optimization peaks when we combine the Open File Cache with two Linux kernel transfer optimization directives: sendfile and tcp_nopush. The three are often called Nginx’s “Static Performance Trio”.

Here’s the complete configuration optimized for a production-level static file server:

http {
    # 1. Open File Cache Configuration
    open_file_cache max=20000 inactive=30s;
    open_file_cache_valid 60s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;

    # 2. Enable Sendfile (Zero-Copy Transfer)
    # Avoids copying file data to Nginx's user space memory
    sendfile on;

    # 3. Enable TCP Nopush (Only active if sendfile is on)
    # Forces Nginx to send the response header and file content in one whole TCP packet
    tcp_nopush on;

    # 4. Enable TCP Nodelay (Bypass the Nagle algorithm)
    # Sends data instantly without delay for keepalive connections
    tcp_nodelay on;

    server {
        listen 443 ssl;
        server_name assets.unisbadri.com;

        root /var/www/assets;

        location / {
            expires 1y;
            add_header Cache-Control "public, immutable";
            access_log off;
        }
    }
}

How Does This Trio Work Together? #

The client requests the logo.png image file.
Nginx checks the open_file_cache in RAM and directly takes the logo.png file descriptor without wasting CPU on the open() syscall.
Nginx calls the sendfile() syscall with that file descriptor. The Linux kernel directly copies the file data from the kernel page cache to the network socket instantly (zero-copy transfer), without copying the data to the Nginx application memory space first.
The tcp_nopush on directive ensures the HTTP response headers and image data are combined and sent together in one maximum TCP network packet (Maximum Segment Size - MSS), minimizing the number of data packets sent and reducing internet network congestion.

Cache Eviction Mechanism, Memory Management, and OS Limits #

To keep RAM usage controlled and efficient, Nginx implements the Least Recently Used (LRU) algorithm to manage the lifecycle of entries inside open_file_cache.

When the maximum entry limit determined by the max parameter is reached, and there’s a new static file that needs to be put into the cache, Nginx doesn’t reject the new entry. Instead, Nginx scans the cache and evicts the entries least frequently or longest unaccessed by clients.

RAM Structure and Memory Consumption #

Each entry stored by open_file_cache in RAM consists of the following components:

File Descriptor (FD): An integer reference to the open file at the kernel level.
Metadata Information: File size, last modification time (mtime), and the file’s inode on disk.
Access Status: File access frequency information for LRU calculations.
Error Information: If open_file_cache_errors is active, access failure details (like the ENOENT code for file not found or EACCES for permission problems) are also recorded.

On average, one entry takes about 500 bytes to 1 kilobyte of RAM. With the max=20000 configuration, Nginx only consumes about 10 to 20 Megabytes of RAM. This is a very cheap resource investment compared to the I/O throughput performance gain we get.

Relationship with Operating System (OS) Limits #

One crucial thing often overlooked by system administrators is the maximum file descriptor limit allowed by the Linux operating system. Every open file entry in open_file_cache uses one actual file descriptor from the system allocation.

If we set open_file_cache max=50000;, but our Linux system limits the file descriptor count per worker process to 1,024 (the Linux default for non-root processes), Nginx will hit the too many open files error and start rejecting new client connections.

Therefore, we must make sure the worker_rlimit_nofile directive in our global Nginx configuration is set larger than the total combined needs of connections and open file cache:

# At the outermost part of the nginx.conf configuration (global context)
worker_rlimit_nofile 65535; # Set the fd limit as high as possible

Field Verification Guide Using strace #

How can we be truly sure that Nginx has stopped making system calls to the operating system after open_file_cache is enabled? We can verify it directly using a Linux process analysis utility called strace.

The steps to do the verification on our test server are as follows:

1. Find the PID of the Nginx Worker Process #

We need to monitor the worker process, not the master process, because the worker handles file delivery to clients. Run the following command:

ps aux | grep nginx

The output will show the master process (running as root) and one or several workers (running as nginx or www-data). Note the PID of one of those worker processes.

2. Run strace Before Enabling the Cache #

Use strace to record file opening syscalls on that worker process while we send several HTTP requests using curl or a performance testing tool like wrk:

# Replace <worker_pid> with our actual worker PID
sudo strace -p <worker_pid> -e trace=openat,fstat,close

Send several requests to fetch the same static file:

curl -I http://localhost/assets/styles.css
curl -I http://localhost/assets/styles.css

In the strace terminal screen, we’ll see syscall logs scrolling for every incoming request:

openat(AT_FDCWD, "/var/www/assets/styles.css", O_RDONLY|O_NONBLOCK) = 12
fstat(12, {st_mode=S_IFREG|0644, st_size=12045, ...}) = 0
close(12)                               = 0

The log above proves that without the cache, the operating system is continuously forced to open, read the status, and close the same file descriptor repeatedly.

3. Run strace After Enabling the Cache #

Now, enable open_file_cache in our Nginx configuration, reload the configuration with nginx -s reload, then find the new worker PID (because the reload triggers new worker processes). Run the strace command again:

sudo strace -p <new_worker_pid> -e trace=openat,fstat,close

Send the same requests consecutively again. On the first request (cache miss), we’ll still see the openat and fstat syscalls. However, on the second, third, and subsequent requests (cache hits), our strace terminal will be silent and show no openat or fstat syscalls at all.

This empirically proves that Nginx serves files entirely from its RAM descriptor cache without touching Linux filesystem syscalls.

Special Scenario: Network File System (NFS) and Distributed Storage #

If our Nginx server reads data from a modern local storage system based on Solid State Drives (SSD) with NVMe connections, the latency for open() and stat() calls might only be under one millisecond. However, the situation changes drastically when we use a Network File System (NFS) or other distributed filesystems (like GlusterFS or CephFS).

In modern-scale web architecture, it’s common to put static assets on centralized storage servers and share them to several Nginx load balancer nodes via NFS mounting.

Network Overhead on Filesystem Syscalls #

When Nginx doesn’t use open_file_cache in an NFS environment:

Every time a client requests a file, the open() syscall triggers the local Linux kernel to send a network request packet to the remote NFS server.
The remote NFS server looks for the file on its own disk, opens the file descriptor, and sends the response back over the network.
The same process happens for the fstat() syscall.

This means one simple static asset request in Nginx requires several network round-trips just to open the file, even before the file data starts being transferred. This inter-server network latency can slow down the HTTP response time (Time to First Byte - TTFB) by tens or hundreds of milliseconds.

The Saving Solution from open_file_cache #

By configuring open_file_cache on our Nginx nodes:

Remote file descriptors are stored in the local RAM of our Nginx server.
Nginx only needs to verify file changes according to the open_file_cache_valid interval (e.g., every 60 seconds) to minimize network load.
All daily file access is served instantly without generating additional network traffic between the Nginx node and the NFS server.

In real-world testing, enabling open_file_cache on an NFS-based Nginx setup can increase static file delivery throughput by 300% to 500% while significantly reducing internal network bandwidth usage.

Stale Cache Problems and Release Strategies #

Although it provides huge performance improvements, using open_file_cache stores one main operational trap: Stale Cache.

If we update static file content on our server (e.g., overwriting the banner.png image file with a new promotion image), clients accessing that file might still receive the old image, or even experience errors if the new file has a different size but Nginx still uses the old metadata size information stored in the cache memory.

To prevent this stale file problem from ruining our user experience, we can apply several of the following strategies:

1. Using Unique File Names (Asset Hashing) #

This is the industry-standard best practice in the modern web era. Our frontend build flow (using Vite, Webpack, or Next.js) should be configured to add a unique hash of the file content to the final file name (e.g., main.a8f9c2d1.js instead of main.js).

Because every code change inevitably changes the file name, Nginx detects it as a new file (Cache MISS) and opens a new file descriptor. Old files no longer in use are automatically evicted from open_file_cache memory after the inactivity period (inactive).

2. Manually Reloading Nginx During Deployment #

If we don’t use asset hashing techniques and are forced to overwrite the same static file directly on disk, we must instruct Nginx to instantly flush its RAM cache.

We can do this without cutting active user connections by sending a reload signal to Nginx:

sudo nginx -s reload

The reload signal makes the Nginx master process create new worker processes with memory clean of old cache entries, then gracefully stops the old worker processes after they finish serving the active connections in progress.

3. Optimizing open_file_cache_valid #

If we can’t manually reload Nginx on every deployment, we must lower the open_file_cache_valid value to something shorter (e.g., 5s or 10s). This forces Nginx to check disk file changes more often. Although it slightly increases CPU load, this step ensures users won’t see old files too long after a new deployment.

Use Case Analysis: When Is This Feature Beneficial? #

The Open File Cache is very effective, but its benefits heavily depend on our application architecture:

Very Beneficial When: #

Static Asset Servers: Custom CDN servers or file storage servers (like MinIO or Vite/React static asset servers) serving thousands of CSS, JS, image, and font files directly from local disk.
High-Concurrency Sites: Servers serving tens of thousands of simultaneous visitors at once.
Slow Storage Systems: Servers whose files are stored on network-attached storage (Network Attached Storage / NFS) or cloud block storage systems (like AWS EBS) with slow filesystem lookup latency.

Less Beneficial When: #

Pure Reverse Proxy: If our Nginx only acts as an entry gate (API Gateway) where 100% of traffic is directly forwarded to upstream backends (like Node.js or Docker containers). Nginx never reads local files from its own server disk for those requests, so the descriptor cache will never fill up.

Summary and Best Practices #

Use max Proportionally: Set the max parameter slightly higher than the total number of active static files in our web server repository.
Enable open_file_cache_errors: Always turn on error caching to cut disk I/O load caused by non-existent file request scans (404).
Combine with Sendfile: Always enable sendfile on; and tcp_nopush on; together with the open file cache to achieve the highest data transfer speed.
Adjust Validation: Set open_file_cache_valid lower (e.g., 10-20 seconds) if we often deploy new assets to the server without changing the file name hashes.
Avoid on Pure Reverse Proxies: Turn off or ignore this configuration if our Nginx server purely acts as a reverse proxy bridge to backends.

← Previous: Keepalive Next: Built-in Modules →