VMware ESXi and vSphere Cluster Management

Rotate, Compress, and Retain Linux Logs with logrotate

Learn how to configure, test, schedule, compress, retain, and safely rotate Linux logs with logrotate, including service reloads, copytruncate, and troubleshooting.

Log files are essential for troubleshooting, auditing, and understanding what services are doing. They are also usually appended continuously. Without a retention policy, a busy log can eventually consume all available disk space.

logrotate is the standard Linux utility for managing this growth. It can rename active logs, create replacement files, compress older generations, remove logs beyond a retention limit, and optionally mail logs before they expire.

Why log rotation is necessary

An active log is the current file to which a service or application writes. If that file grows forever, the filesystem may run out of capacity or inodes.

A full filesystem can cause services to fail, prevent applications from writing data, break package operations, and make databases or queues stop unexpectedly. It can also destroy the diagnostic information needed to investigate the incident because new error messages cannot be written.

Rotation preserves a limited history while replacing or reopening the active log. A typical lifecycle is:

1. The application writes to an active file such as /var/log/example-app/app.log.
2. logrotate renames that file to an older generation.
3. logrotate creates a new active file with suitable ownership and permissions.
4. The service reopens the new file, or logrotate uses a compatibility method such as copytruncate.
5. Older generations are compressed and removed when the retention limit is exceeded.

What logrotate does

logrotate applies rules to one or more matching log paths. A rule specifies when rotation is due and what should happen afterward. Global defaults can provide common behavior, while a log-specific block can replace those defaults for one service or application.

Rotation is evaluated when logrotate runs. The daily directive does not create its own process or timer; a cron job, anacron job, or systemd timer must invoke logrotate.

Configuration layout

Main configuration and includes

The primary configuration file is /etc/logrotate.conf. It commonly contains global defaults and an include directive that loads additional policies.

# Global defaults belong here
weekly
rotate 4

include /etc/logrotate.d

/etc/logrotate.d/ is the conventional directory for package-specific and locally managed policies. For example, a web server package may install its own file there.

Settings in a log-specific block can replace applicable global defaults. This lets one busy application use daily rotation and compression even when the global policy is weekly. Avoid creating a second policy for a package-managed log unless you have checked the existing file; conflicting rules can cause surprising rotations.

Inspect the effective configuration sources before adding a policy:

sudo sed -n '1,220p' /etc/logrotate.conf
sudo ls -la /etc/logrotate.d
sudo sed -n '1,220p' /etc/logrotate.d/<service-name>

Logrotate rule syntax

A rule starts with one or more log paths and ends with a braced block of directives. Paths may include wildcards when a group of related files should share one policy. Comments begin with #; whitespace is used to make rules readable.

/var/log/example-app/app.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
}

Multiple paths can share one block:

/var/log/web/access.log /var/log/web/error.log {
    weekly
    rotate 8
    compress
    sharedscripts
    postrotate
        systemctl reload web-service.service
    endscript
}

The closing brace ends the rule. Script blocks end with endscript. Keep locally created files clearly named and add comments explaining unusual thresholds or service commands.

Rotation schedules and triggers

Time-based schedules

The common time directives are:

• daily: evaluate the log for daily rotation.
• weekly: evaluate it weekly.
• monthly: evaluate it monthly.
• yearly: evaluate it yearly.

The exact result depends on when the scheduler invokes logrotate and on the state file, which records previous rotations. A weekly rule cannot rotate on schedule if logrotate is never run.

Size-based rotation

size rotates when the log reaches a threshold. Units can be expressed as bytes, kilobytes with k, megabytes with M, or gigabytes with G.

/var/log/example-app/app.log {
    size 10M
    rotate 7
    compress
}

minsize is different. It adds a minimum-size condition to a time interval. For example, weekly with minsize 50M means the weekly check is eligible to rotate only if the file is at least 50 MB. This avoids creating many tiny archives while still enforcing a weekly opportunity to rotate.

Rotation trigger comparison

Time interval: Best for predictable history such as daily web logs. It depends on scheduler frequency. A daily policy produces an opportunity on each invocation after the relevant interval.

Maximum size: Best for high-volume logs. It depends on scheduler frequency, so a file may exceed the threshold before the next check.

Time interval plus minimum size: Best when archives should be time-based but empty or tiny logs should not be rotated. Both the interval opportunity and minimum size condition matter.

Choose a schedule according to log volume, available disk capacity, the period needed for incident investigation, and compliance requirements. High-volume logs may need frequent scheduler invocation or application-level controls in addition to logrotate.

Retention and removal

rotate followed by a count specifies how many older generations to retain. For example, rotate 7 keeps seven older archives in addition to the active file. When a new generation would exceed that limit, the oldest generation is removed.

/var/log/service/service.log {
    daily
    rotate 14
}

Retention is a capacity and risk decision. Consider disk space, the time required to investigate incidents, legal or compliance requirements, backups, and whether copies are sent to centralized logging. Local rotation should not be treated as the only preservation mechanism.

Naming rotated logs

Without date-based naming, generations commonly receive numbered suffixes such as .1 and .2. The newest older generation is usually numbered lowest, with older files moving to higher numbers as rotation proceeds.

dateext uses a date suffix instead, such as app.log-2026XXXX, depending on the date format and implementation. Date-based names can make searching, auditing, and correlating a file with an incident easier.

/var/log/example-app/app.log {
    daily
    rotate 7
    dateext
    dateformat -%Y-%m-%d
}

dateformat is supported by common logrotate versions, but consult the installed version when relying on a particular format. If more than one rotation can occur on the same day, include enough time information or another uniqueness strategy so that filenames cannot collide. A date suffix alone may not distinguish multiple daily rotations.

Compression

compress compresses archived logs, normally with gzip, reducing disk usage. Compression is especially useful when retaining several weeks or months of history.

• compress: compress archived generations.
• delaycompress: leave the newest rotated file uncompressed until the next rotation, then compress it.
• nocompress: disable compression when immediate readability or another operational requirement outweighs disk savings.

delaycompress can help services that briefly continue reading the immediately previous file or tools that expect the newest archive to remain plain text. Compression trades CPU and I/O during rotation for lower storage usage. Match the policy to log volume, available CPU, retention duration, and how often operators need to inspect archives.

Creating and handling active log files

The create directive

create makes a replacement active log after rotation. Its arguments can specify the permission mode, owner, and group:

create 0640 appuser appgroup

The replacement file must be writable by the process that produces the log. A wrong owner, group, or mode can cause permission-denied errors immediately after rotation. Parent-directory permissions and security controls such as SELinux or AppArmor also matter.

Optional and empty logs

missingok prevents an absent optional log from producing a routine error. notifempty skips rotation when the file is empty, avoiding needless empty archives.

/var/log/optional-service/events.log {
    weekly
    rotate 4
    missingok
    notifempty
}

The su directive

su tells logrotate which user and group to use when operating on logs in directories or files owned by non-root users. Use it when the directory layout requires a non-root identity, and verify that the selected account can perform the required rename, creation, and compression operations.

Service-aware rotation

Why a reload may be required

Renaming a file does not automatically change a daemon's open file descriptor. A service can continue writing to the old inode even though the pathname now points to a new file. The result is an apparently empty active log while disk usage continues growing in the renamed file.

The preferred solution is to create a new file and signal or reload the service so it reopens its logs. The command depends on the application.

prerotate, postrotate, endscript, and sharedscripts

prerotate contains shell commands run before rotation. It can validate a condition or prepare a service. postrotate contains commands run after rotation, commonly a reload or reopen-log signal. Each script block ends with endscript.

When a rule contains several paths, script blocks may run once per matched file. sharedscripts changes this behavior so the script runs once for the whole rule, which is usually appropriate for one service reload.

/var/log/web/access.log /var/log/web/error.log {
    weekly
    rotate 8
    compress
    delaycompress
    sharedscripts
    postrotate
        systemctl reload web-service.service >/dev/null 2>&1 || true
    endscript
}

Use the real service's documented reload or log-reopen operation. Test it independently before relying on the rotation hook.

copytruncate

copytruncate copies the active file to an archive and then truncates the original file in place. The application can keep its existing file descriptor and continue writing without a reload.

/var/log/legacy-app/service.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

This is a fallback for applications that cannot reopen logs. Copying and truncating are not atomic with respect to writers, so a small window can lose or duplicate entries. Prefer a service reload or application-managed logging for busy or high-value logs.

Service log handling choices

Create a new file and signal or reload: The service closes the old descriptor and opens the replacement. This preserves normal file semantics and is usually preferred. It requires a reliable service command.

copytruncate: The service continues using the original pathname and descriptor. It is compatible with legacy applications, but copying can lose or duplicate entries during the transition.

Application-managed logging or journald: The application or logging system controls file lifecycle. This can provide richer policies, but its own retention and storage settings must be configured and monitored.

Mailing old logs

The mail directive sends a rotated log to a specified address when that log is about to be removed because it has exceeded the retention count. mailfirst selects the newly rotated log for mailing, while maillast selects the oldest relevant log before deletion.

/var/log/security/events.log {
    weekly
    rotate 8
    compress
    mailfirst
    mail admin@example.invalid
}

Delivery requires a functioning local mail transport configuration. Before enabling this feature, consider that logs may contain credentials, personal data, tokens, internal addresses, or other sensitive information. Use an approved secure delivery path, access-controlled mailbox, and appropriate data-handling policy.

Scheduling logrotate

Depending on the Linux distribution, logrotate is normally launched by cron, anacron, or a systemd timer. The scheduler frequency determines how accurately time and size policies are enforced.

• A daily invocation can detect a daily policy once per day.
• A weekly policy may be evaluated during each invocation, with the state file preventing premature repeated rotation.
• A size threshold can be exceeded for hours if logrotate runs only once per day.
• High-volume logs may require more frequent invocation, application rate controls, or centralized collection.

systemctl list-timers --all | grep -i logrotate
systemctl status logrotate.timer
sudo ls -la /etc/cron.daily | grep logrotate

Unit names and cron locations vary by distribution. Check the scheduler actually used by the host rather than assuming a particular mechanism.

Running and validating logrotate manually

Debug and verbose modes

Use debug mode to inspect planned decisions without changing files:

sudo logrotate --debug /etc/logrotate.conf

Verbose mode reports decisions and rotation steps:

sudo logrotate --verbose /etc/logrotate.conf

Always test a new rule before depending on automated execution. Confirm the path, trigger, retention count, ownership, permissions, names, compression result, and service behavior.

Forced testing

Force a controlled rotation with:

sudo logrotate --force --verbose /etc/logrotate.conf

Use force cautiously on production systems. It can create an extra generation and execute service hooks even when the normal schedule says rotation is not due.

The state file

logrotate uses a state file to record prior rotations. The state helps determine whether a daily, weekly, monthly, or yearly policy is due. If a test appears not to rotate, the state file may show that the log was already processed. Prefer debug output and normal controlled testing over casually deleting or editing state data.

Practical policy examples

Small custom application log

Place a locally managed policy in /etc/logrotate.d/. This example rotates at 10 MB, keeps seven archives, uses date-based names, compresses old generations, and creates a replacement file for the application account.

/var/log/example-app/app.log {
    size 10M
    rotate 7
    compress
    dateext
    missingok
    notifempty
    create 0640 appuser appgroup
    postrotate
        systemctl reload example-app.service >/dev/null 2>&1 || true
    endscript
}

Use the actual service reload or reopen-log operation. Choose the threshold, owner, group, and retention values for the environment.

Web access and error logs

Related web logs can share one rule. Several weeks of compressed history may be useful for traffic analysis and incident investigation. sharedscripts prevents multiple reloads when both files rotate.

/var/log/web/access.log /var/log/web/error.log {
    weekly
    rotate 8
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        systemctl reload web-service.service >/dev/null 2>&1 || true
    endscript
}

Common log locations

System messages: Commonly under /var/log/, produced by the system logging service or distribution-specific components. Check existing package policies before adding one.

Authentication events: Often stored in a security or authentication log under /var/log/. Protect permissions and consider compliance retention.

Package or service logs: Often under /var/log/ or a service-specific subdirectory. Package-provided files in /etc/logrotate.d/ may already manage them.

Web access logs: Commonly in a web-server directory under /var/log/. They can grow quickly and usually benefit from compression.

Web error logs: Commonly beside access logs. Coordinate rotation with a reload or reopen signal.

Custom application logs: Often placed in a dedicated directory under /var/log/. Set explicit ownership and permissions matching the application account.

Reviewing rotation results

sudo ls -lh /var/log/example-app/
sudo du -sh /var/log/example-app/
sudo zcat /var/log/example-app/app.log-<date>.gz | less

Check that the active file exists, the writer can append to it, archives have the expected names, compression is present when intended, and the number of generations matches the policy.

Troubleshooting logrotate

The log was not rotated

• Check whether the cron job or systemd timer ran.
• Run debug and verbose modes.
• Check whether the time or size condition has been met.
• Inspect the applicable rule and state file.
• Confirm that the tested configuration includes the intended file.

The service writes to the archived file

The daemon probably retained an open descriptor to the old inode. Identify the process with a tool such as lsof, verify the documented reopen or reload action, and test the postrotate command independently.

The new active log has permission errors

Compare the service account with the create owner, group, and mode. Also inspect parent-directory permissions and relevant SELinux or AppArmor logs and labels.

Archives are not compressed

Check whether the active rule contains compress or an overriding nocompress. If delaycompress is present, the newest archive is intentionally left uncompressed until the next cycle. Also verify that the configured compressor is installed and usable.

Configuration parse errors

Common causes include missing braces, an unterminated script block, misspelled directives, invalid arguments, or unsafe configuration-file permissions. Debug mode usually identifies the failing file and line.

Size rotation occurs late

logrotate cannot enforce a threshold between invocations. Check scheduler frequency and increase it when appropriate. Extremely busy logs may also need application-level limits or centralized collection.

Email is not received

Check for a functional local mail transfer agent, recipient and relay configuration, mail queues, local mail logs, message-size limits, and security-policy rejections. Test mail delivery separately and reassess whether sending raw logs by email is appropriate.

Operational design guidance

• Use frequent rotation and compression for high-volume web or application logs.
• Use longer retention where incident investigation or compliance requires it.
• Set explicit create ownership and permissions for custom application logs.
• Prefer service reload or reopen support over copytruncate.
• Do not duplicate a package's existing policy without understanding precedence and conflicts.
• Use clear local configuration filenames and comments.
• Align local retention with backups and centralized logging so important records survive host failure.
• Monitor filesystem capacity and inode usage in addition to configuring rotation.

Exam-relevant notes

• daily, weekly, monthly, and yearly are evaluated when logrotate is invoked; they are not independent schedulers.
• size is a size trigger, while minsize adds a minimum-size condition to a time-based policy.
• rotate 7 means seven older generations are retained.
• compress normally uses gzip; delaycompress postpones compression of the newest archive.
• A daemon may keep writing to a renamed file unless it reopens its descriptor.
• copytruncate avoids a reload but can lose or duplicate entries during copying.
• missingok handles absent files; notifempty skips empty files.
• The state file records previous rotations and affects whether a time-based rotation is due.

For related learning, see Linux log rotation.