Custom Log Formats #
The default combined log format used by Nginx is fine for basic needs, but as our application infrastructure grows in complexity, the need for deeper information becomes a must. We often need data about how fast the backend responds to requests, whether content is served from a local cache or has to be fetched again from upstream, and how to trace a transaction from the user’s browser all the way to the deepest microservices.
Fortunately, Nginx is designed with a very high degree of configuration flexibility. Using the log_format directive, we can design any information recording pattern supported by Nginx’s internal variables. In this article, we’ll discuss how to define custom log formats, record performance metrics precisely, compose valid JSON formats for modern log aggregators using the escape=json parameter, and implement distributed tracing using $request_id.
Basic Syntax of the log_format Directive
#
The log_format directive can only be declared inside the global http context. We can’t write this directive directly inside a server or location block. However, once the format is declared at the http level, we can use it in any server block by calling its name in the access_log directive.
Writing Syntax #
http {
log_format format_name [escape=default|json|none] format_string;
}
format_name: The custom unique identifier name we’ll call in theaccess_logdirective. We can’t use built-in system format names likecombined.escape: Determines the escaping method for special characters (like quote characters, tabs, or newlines) found inside variable values. The best choice for plain text formats isdefault, while for JSON log formats we must usejson.format_string: The collection of text and Nginx variables that make up the log line pattern to be generated.
Here’s an example of a simple declaration and its usage:
http {
# Declare a custom format named 'simple'
log_format simple '$remote_addr - $status - $request';
server {
listen 80;
server_name example.com;
# Call the custom format in the server block
access_log /var/log/nginx/example_simple.log simple;
}
}
Adding Timing Variables for Performance Monitoring #
One of the most important customization scenarios in production environments is recording timing metrics. With this timing data, we can objectively solve web application performance degradation problems.
Nginx provides several crucial time variables we can include in the log format:
$request_time: The total request processing time from the first byte received from the client to the last response byte sent to the client. This time is measured in seconds with millisecond precision. This time includes data transmission over the network, so if the client’s internet connection is slow, this variable’s value will be high.$upstream_response_time: The time the backend server (upstream) took to process the request and send the response back to Nginx. This variable purely reflects our application’s performance (like PHP, Node.js, Go) without being affected by the client’s internet speed.$upstream_connect_time: The time Nginx took to make the network connection (TCP handshake) with the backend server. If this value is high, there’s likely internal network congestion or the backend load is too high to accept new connections.$upstream_header_time: The time between the start of the connection to the backend and when Nginx receives the first byte of the response header from the backend.
Example Performance Log Format #
We can put together a special log format to test server performance like this:
http {
log_format performance '$remote_addr [$time_local] "$request" '
'status=$status bytes=$body_bytes_sent '
'req_time=$request_time up_resp_time=$upstream_response_time '
'up_conn_time=$upstream_connect_time cache=$upstream_cache_status';
server {
listen 80;
server_name api.unisbadri.com;
# Use the performance format to track latency
access_log /var/log/nginx/api-performance.log performance;
}
}
Reading Performance Logs for Latency Analysis #
Suppose we see the following log entry in the api-performance.log file:
203.0.113.12 - [16/Jun/2026:13:30:00 +0700] "GET /api/data HTTP/1.1" status=200 bytes=1048576 req_time=4.120 up_resp_time=0.105 up_conn_time=0.002 cache=MISS
From the line above, we can analyze the following performance conditions:
up_resp_time=0.105: The backend processed the data very quickly, only taking 105 milliseconds.req_time=4.120: However, the total request completion time was 4.120 seconds.- Analysis: There’s a time difference of about 4 seconds. Because the data size sent is quite large (
bytes=1048576or 1 MB), we can conclude that our backend is healthy, but the client’s internet connection is slow at downloading that 1 MB response data.
JSON Formats for Modern Log Aggregators #
In modern cloud computing infrastructure, we rarely read log files manually directly on individual servers. We usually send log data to centralized log collection systems (log aggregators) like the ELK Stack (Elasticsearch, Logstash, Kibana), Grafana Loki, Datadog, Splunk, or AWS CloudWatch.
Those log aggregator tools really like the JSON format because of its organized key-value pair structure. This makes indexing, searching, and building visualization dashboards easier without needing to write complicated regex parser rules.
The Urgency of the escape=json Parameter
#
When creating a JSON log format, we must add the escape=json parameter to the log_format declaration.
If we don’t enable it, special characters like double quotes (") in user agents or request paths sent by clients will be written as-is to the log file. This will break the JSON syntax structure (causing invalid JSON format) and make log aggregators fail to read that data.
With escape=json, Nginx automatically converts special characters into safe strings (e.g., changing " to \" or \ to \\) so the JSON structure is guaranteed to always be valid.
Production JSON Format Configuration #
Here’s a highly recommended standard production JSON log format configuration:
http {
log_format json_combined escape=json
'{'
'"timestamp":"$time_iso8601",'
'"client_ip":"$remote_addr",'
'"request_id":"$request_id",'
'"method":"$request_method",'
'"scheme":"$scheme",'
'"host":"$host",'
'"uri":"$uri",'
'"query_string":"$args",'
'"status":$status,'
'"bytes_sent":$body_bytes_sent,'
'"request_time":$request_time,'
'"upstream_response_time":"$upstream_response_time",'
'"upstream_connect_time":"$upstream_connect_time",'
'"upstream_cache_status":"$upstream_cache_status",'
'"referrer":"$http_referer",'
'"user_agent":"$http_user_agent",'
'"x_forwarded_for":"$http_x_forwarded_for"'
'}';
server {
listen 443 ssl;
server_name app.unisbadri.com;
# Save logs in JSON format
access_log /var/log/nginx/app-access.json json_combined;
}
}
Distributed Tracing using $request_id
#
In microservices architecture, one request from a user’s browser can trigger a chain of API calls to a dozen different internal backend services. If a failure occurs in one service in the middle of that chain, tracing the cause becomes very difficult because each service logs its own records.
To overcome this challenge, we use the concept of Distributed Tracing. Nginx, as the main entry gate (reverse proxy / API gateway), acts as the party that creates a unique identifier for every incoming request, then forwards that identifier to the entire backend system.
The $request_id Variable
#
Nginx provides a built-in variable called $request_id (available since Nginx version 1.11.0). This variable generates a unique random hexadecimal string value of 32 characters for every processed request.
Here’s an illustration of the distributed tracing flow using the request ID:
flowchart TD
Client["Browser Client"] -->|"1. Send Request"| Nginx["Nginx Reverse Proxy<br/>(Generate $request_id)"]
Nginx -->|"2. Log with request_id"| LogNginx["Nginx Access Log"]
Nginx -->|"3. Add X-Request-ID header"| Backend["Backend API Service<br/>(Node.js / Go)"]
Backend -->|"4. Request Data"| DB["Database Server"]
Backend -->|"5. Log with request_id"| LogBackend["Backend App Log"]
classDef default fill:#f9f9f9,stroke:#d1d5db,stroke-width:1px,color:#111827;
classDef highlight fill:#eff6ff,stroke:#3b82f6,stroke-width:2px,color:#1e3a8a;
class Nginx,Backend highlight;Distributed Tracing Configuration in Nginx #
We must configure Nginx to:
- Record
$request_idin our access log. - Forward
$request_idto the backend server via theX-Request-IDHTTP header. - Return
$request_idto the client browser in a response header so clients can use it as a reference when reporting issues to the support team.
http {
log_format trace_format '$remote_addr [$time_local] '
'req_id=$request_id status=$status '
'rt=$request_time urt=$upstream_response_time';
server {
listen 80;
server_name shop.unisbadri.com;
access_log /var/log/nginx/shop-access.log trace_format;
# 1. Include the Request ID in the response header so client browsers can read it
add_header X-Request-ID $request_id always;
location / {
# 2. Forward the Request ID to the internal backend application
proxy_set_header X-Request-ID $request_id;
proxy_pass http://shop_backend;
}
}
}
On our backend application side (e.g., using PHP, Node.js, or Go), the developer team must configure their application logging libraries to read the X-Request-ID header and include it in every application log line and database query.
That way, if a transaction failure occurs, we can simply copy the Request ID reported by the user’s browser and search for it in the log aggregator. The entire transaction activity from the Nginx level, backend API, to database queries will be correlated instantly using the same unique ID.
Recording Custom HTTP Headers (Request & Upstream) #
Nginx also lets us record data from custom HTTP headers sent by the client as well as those returned by the backend.
- To record a header sent by the client, use the
$http_variable prefix followed by the header name in lowercase, with hyphens (-) changed to underscores (_). - To record a header returned by the backend, use the
$upstream_http_variable prefix with the same writing rules.
Example Header Recording Scenario #
Suppose our application uses the following custom headers:
- The client sends the
X-Client-Versionheader to identify the mobile app version. - The backend returns the
X-Process-Memoryheader to report application memory consumption.
We can record both pieces of information in our Nginx log like this:
http {
log_format app_header_log '$remote_addr - [$time_local] '
'status=$status '
'client_ver=$http_x_client_version ' # Refers to X-Client-Version from the client
'backend_mem=$upstream_http_x_process_memory'; # Refers to X-Process-Memory from the backend
server {
listen 80;
server_name test.example.com;
access_log /var/log/nginx/app-headers.log app_header_log;
location / {
proxy_pass http://app_backend;
}
}
}
Recording these custom headers greatly helps our operations team track specific bugs in a particular mobile app version without needing to change the backend application code.
Filtering Sensitive Data in Logs (Data Masking & Anonymization) #
In the era of strict data protection regulations like Indonesia’s Personal Data Protection Law (UU PDP) or the EU’s GDPR, recording users’ sensitive data into log files is a serious compliance violation. Data like full IP addresses, authorization tokens, passwords in query strings, or credit card numbers must not be stored in plain text in log files accessible to many parties.
We can leverage the power of the map directive in Nginx to filter (mask) or anonymize that data before it’s written to the log file.
1. Client IP Address Anonymization #
To protect user privacy, we can mask the last octet of an IPv4 address or the final segment of an IPv6 address using regex inside the map block:
http {
# Mask the last octet of IPv4 and half of the IPv6 segments
map $remote_addr $ip_anonymized {
# IPv4 example: 203.0.113.88 becomes 203.0.113.0
~^(?P<first>\d+\.\d+\.\d+)\.\d+$ $first.0;
# IPv6 example: 2001:db8:85a3::8a2e:370:7334 becomes 2001:db8:85a3::
~^(?P<first>[0-9a-fA-F:]+:[0-9a-fA-F:]+:[0-9a-fA-F:]+):[0-9a-fA-F:]+$ $first::;
default 0.0.0.0;
}
log_format anonymous_log '$ip_anonymized - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent';
}
By using $ip_anonymized instead of $remote_addr in the log_format declaration, we can still analyze the macro geographic region of our site visitors without storing their specific Personally Identifiable Information (PII).
2. Masking Authorization Tokens (Bearer Token)
#
If our application uses token-based authentication (JWT) sent via the Authorization header, recording that header raw will leak user access credentials. We can mask the token and only keep the first few characters:
http {
# Mask bearer authorization tokens
map $http_authorization $masked_authorization {
# If the header contains "Bearer abcdef12345...", change it to "Bearer abcdef***"
~^Bearer\s+(?P<prefix>.{6}).*$ "Bearer $prefix***";
# If empty, keep it empty
"" "";
# Otherwise, mask it completely
default "***";
}
log_format security_audit_log '$remote_addr [$time_local] '
'status=$status auth="$masked_authorization"';
}
3. Masking Sensitive Parameters in URLs (Query Strings) #
Often, less secure backend applications send access tokens or sensitive data through URL parameters (e.g., /reset-password?token=secret123). To prevent these tokens from being recorded in the $request or $args variables, we can define a custom URL filter:
http {
# Clean secret tokens from the query string
map $request_uri $filtered_request_uri {
# Find 'token=...' or 'password=...' parameters and replace their values with [REDACTED]
~^(?P<path>[^?]+)\?(?P<query1>.*)(?:token|password)=[^&]+(?P<query2>.*)$ $path?$query1token=[REDACTED]$query2;
# If it doesn't contain sensitive parameters, use the original URL
default $request_uri;
}
log_format safe_url_log '$remote_addr - [$time_local] "$request_method $filtered_request_uri" $status';
}
Log Aggregator Integration (Grafana Loki & Filebeat) #
Collecting logs on the local server disk is only the first step. For large-scale production infrastructure, we must send those logs to a log aggregator. There are two main architectural approaches for sending logs from Nginx:
Approach 1: Log Shipper Agent (Industry Recommendation) #
In this approach, Nginx still writes logs in JSON format to local files on disk. Then, a log shipper agent application like Promtail (for Grafana Loki), Filebeat (for Elasticsearch), or Vector runs in the server background to tail changes to those log files and send them to the central server.
flowchart LR
A["Nginx JSON<br/>Log Engine"] -->|"Writes to"| B["Log File<br/>on Disk"]
C["Log Shipper Agent<br/>(Promtail/Filebeat)"] -->|"Reads"| B
C -->|"Sends"| D["Central Aggregator<br/>(Loki/Elasticsearch)"]The advantage of this system is high reliability. If the central log server is experiencing downtime, the log shipper agent records the last file read position and will resend the remaining log queue after the central server comes back online (backpressure handling).
Approach 2: Direct Delivery via the Syslog Protocol #
If we want to avoid using local disk I/O entirely for performance or disk security reasons, Nginx supports sending log entries directly using the Syslog protocol over UDP or TCP networks:
http {
# Send logs directly to the centralized log server via Syslog UDP port 514
access_log syslog:server=10.0.0.50:514,facility=local7,tag=nginx,severity=info json_combined;
}
server=10.0.0.50:514: Specifies the IP address and port of the Syslog receiving server.facility=local7: The Syslog facility category used (generallylocal0tolocal7for custom applications).tag=nginx: An identification tag to make log filtering easier on the receiving server.severity=info: Determines the severity level of log message delivery.
The drawback of the UDP Syslog approach is the lack of delivery guarantees (fire-and-forget). If our internal network experiences congestion, some log entries may be lost along the way without Nginx detecting it.
Summary and Best Practices #
- Use escape=json for JSON Logs: Make sure the
escape=jsonparameter is always active when defining JSON log formats so logs aren’t corrupted by special characters.- Monitor Time Differences: Always include the
$request_timeand$upstream_response_timevariables to measure network performance vs our backend application’s performance.- Implement Request IDs: Use the
$request_idvariable for distributed tracing to instantly unify Nginx log correlation with backend application logs.- Anonymize PII (Personally Identifiable Information): Use the
mapdirective to mask client IP addresses and censor sensitive authentication tokens before writing to disk.- Group Declarations at http: All
log_formatdirectives must be written inside the globalhttpblock, not insideserverorlocationblocks.