VMware ESXi and vSphere Cluster Management

rsyslog: Linux System Logging Configuration and Remote Log Forwarding

Learn how rsyslog receives, filters, stores, broadcasts, and forwards Linux log messages using facilities, priorities, selectors, actions, and configuration files.

rsyslog is a modular Linux logging daemon that is compatible with the traditional syslog message model. The daemon process is commonly named rsyslogd. Many Debian-family and Red Hat-family systems install rsyslog as a primary system logging service, although the exact default setup varies by distribution and package version.

Applications, background services, the Linux kernel, and even other computers can submit log records to rsyslog. The daemon evaluates each record against configured rules, then delivers matching records to one or more destinations. Destinations can include local files, terminal devices, logged-in users, or remote log collectors.

What rsyslog does

The traditional syslog model defines a message classification system and a way to transport log messages. rsyslog extends that model with modular inputs and outputs, flexible filtering, multiple destinations, and centralized logging capabilities.

A typical local message flow is:

1. A program, service, or the kernel generates an event.
2. The event is submitted to rsyslog through a local logging interface or an input module.
3. rsyslogd classifies the message by facility and priority.
4. Configured selectors and filters are evaluated.
5. Each matching action writes, broadcasts, or forwards the message.

A message is not necessarily consumed by only one rule. If several rules match, the same record can be written to multiple files and forwarded to a remote host at the same time.

Inputs, processing, and outputs

rsyslog functionality is divided into modules. A module is a component that supplies an input, output, parser, filter, or another capability. Input modules receive records, while output modules send records to destinations. This modular design lets an installation accept local messages, network syslog traffic, or other supported inputs and route them to files, databases, remote systems, or other outputs.

Configuration layout

Configuration files are commonly split so that the primary file handles service-wide setup while smaller files hold related rules. A split configuration is easier to review, troubleshoot, and preserve during package updates. The main file may include a directory or filename pattern with an include directive such as the legacy $IncludeConfig syntax.

Do not assume that a particular numbered filename exists on every system. Inspect /etc/rsyslog.conf and the contents of /etc/rsyslog.d/ before adding a rule.

Modules and enabled lines

Legacy configurations load modules with directives such as:

$ModLoad imuxsock
$ModLoad imklog

A line beginning with # is a comment and is not active. Thus, commenting out a module-load line commonly disables that feature. Modern rsyslog configurations may instead use RainerScript syntax:

module(load="imuxsock")

Available module names, directives, and preferred syntax depend on the installed rsyslog release and package. Do not copy a directive from another system without checking whether that version supports it.

Global directives

Global directives affect service-wide behavior rather than one individual rule. Common legacy directives include:

• $FileOwner controls the owner assigned to newly created log files.
• $FileGroup controls the group assigned to newly created log files.
• $IncludeConfig incorporates configuration files from a path or pattern.

For example, a configuration might contain:

$FileOwner root
$FileGroup adm
$IncludeConfig /etc/rsyslog.d/*.conf

These settings do not by themselves fix every permission problem. The parent directory must exist, the rsyslog process must be able to create or open the file, and mandatory access control policies may also restrict writes.

Traditional rule syntax

A traditional rsyslog rule has two main parts:

selector    action

The selector decides which records match. The action specifies what rsyslog does with matching records. The parts are normally separated by whitespace, and comments or disabled directives typically begin with #.

Facilities

A facility is a category assigned according to the source or class of a message. This lets administrators route kernel messages separately from mail or application messages.

• kern represents kernel messages.
• mail represents mail-system messages.
• * in the facility position represents all facilities.

Other standard facilities exist, including categories commonly used by authentication, system daemons, user programs, and local-use applications. Select the facility that the originating service actually uses.

Priorities and severity matching

A priority, also called a severity, expresses how urgent a message is. The traditional severity ordering runs from the most urgent level, emerg, through progressively less urgent levels. Important levels for this lesson are:

In traditional selector syntax, naming a priority generally selects that priority and all more critical priorities. Therefore, kern.alert matches kernel messages at alert and emerg severity. It does not mean “alert only.” The wildcard priority * matches all priorities for the selected facility.

Local logging actions

Write critical kernel messages to a file

kern.alert    /var/log/kern.log

Here, kern is the facility, alert is the threshold, and /var/log/kern.log is the local-file action. The rule writes alert and emergency kernel records to that file.

The destination directory must exist, and its ownership and permissions must allow rsyslog to create or write the file. Check the configured $FileOwner and $FileGroup, filesystem permissions, disk space, and any mandatory access control policy before using a custom path.

Store all mail messages separately

mail.*    /var/log/mail

The mail facility is selected and the wildcard priority matches every mail-facility severity. This creates a dedicated destination for mail-related records, assuming the parent directory and permissions are suitable.

Notify logged-in terminal users

*.emerg    *

The first wildcard selects every facility, while emerg selects emergency messages. The action wildcard sends matching records to currently logged-in users' text terminals. Delivery to graphical sessions varies by desktop, terminal, and operating-system environment, so this action should not be treated as a universal desktop notification mechanism.

Remote log forwarding

Forwarding sends selected records from one rsyslog host to a remote collector. The selector can remain the same as for local storage; only the action changes.

kern.alert    @suse1

This forwards alert-and-more-severe kernel messages to the host named suse1 using the traditional single-@ UDP notation.

kern.alert    @@loghost

In legacy action syntax, the double-@ notation conventionally selects TCP. The receiving server must be configured to listen for network syslog traffic using the same transport and appropriate port.

UDP and TCP considerations

• UDP: low overhead, but delivery is not reliable. Packets can be lost, reordered, or blocked without a transport-level session.
• TCP: provides a connection and more reliable delivery behavior, but requires a listening TCP service and can be affected by connection failures or backpressure.
• TLS: recommended for production environments where log confidentiality, integrity, authentication, or protection against interception matters. TLS configuration is more involved than the legacy @ and @@ notation.

Before troubleshooting forwarding, verify that the sender can resolve the destination hostname, route traffic to it, and reach the required port. Check firewall policy on both systems. Confirm that the receiver is listening and that sender and receiver agree on UDP versus TCP. A hostname that resolves incorrectly can send logs to the wrong system or make forwarding appear broken.

Example rule outcomes

Why multiple destinations can receive one record

Suppose a mail message has emergency severity. It matches mail.* because every mail priority is selected, and it also matches *.emerg because the message is emergency level. rsyslog can therefore write it to the mail log and broadcast it to logged-in terminal users. Traditional rules are not automatically mutually exclusive.

Applying and validating configuration changes

Editing a configuration file does not immediately change the running daemon. rsyslog must reload or restart its configuration.

sudo service rsyslog restart
sudo systemctl restart rsyslog
sudo systemctl status rsyslog

The first command uses a legacy service-management interface. The second and third are used on systemd-based systems. Use the interface supported by the host. A restart interrupts the daemon briefly; where supported and appropriate, a reload can apply configuration without a full process restart.

Validate syntax before deployment or immediately after editing when the installed version supports it:

sudo rsyslogd -N1

This performs a configuration check on many rsyslog installations. The exact options and validation behavior can vary by release, so consult the installed system's documentation if the command is unavailable or behaves differently.

After applying a rule, generate a controlled test event:

logger -p mail.alert "rsyslog test message"

Then inspect the configured file or remote collector. This test explicitly uses the mail facility and alert severity, making it suitable for verifying a rule such as mail.*. A test that uses a different facility or severity may not match the rule you are checking.

Troubleshooting rsyslog

A new rule has no effect

• Confirm that rsyslog was reloaded or restarted after the edit.
• Check that the edited file is under an included path, such as an included /etc/rsyslog.d/ pattern.
• Run a supported configuration validation command and inspect service status or journal output for syntax errors.
• Use logger to generate a message with the exact facility and priority expected by the selector.
• Inspect the intended destination rather than assuming that a different default log contains the event.

The logfile is missing or cannot be written

• Verify that the parent directory exists.
• Check file and directory ownership, group membership, and permissions.
• Review $FileOwner and $FileGroup or their modern equivalents.
• Check available disk space and filesystem restrictions.
• Review mandatory access control status and related security logs if the permissions appear correct.

Remote forwarding does not arrive

• Verify destination hostname resolution and network reachability.
• Confirm the receiver is listening for syslog traffic on the selected port and transport.
• Check firewall rules on the sender, receiver, and intervening network.
• Confirm that @ is being used for legacy UDP forwarding or @@ for legacy TCP forwarding.
• Check that the receiver's rules store the incoming messages where you expect.

Emergency notifications do not appear

• Confirm the user is logged in through a compatible text terminal.
• Check terminal messaging permissions and environment-specific restrictions.
• Generate an explicit emergency test record and verify that the rule is loaded.
• Remember that terminal broadcast behavior does not guarantee a graphical desktop notification.

Exam-relevant summary

• rsyslog is a modular, syslog-compatible logging daemon; rsyslogd is its daemon process.
• The primary configuration file is /etc/rsyslog.conf; supplementary files commonly reside in /etc/rsyslog.d/.
• A traditional rule consists of a selector and an action.
• A selector has the form facility.priority.
• kern.alert includes kernel messages at alert and emergency severity.
• mail.* selects every priority for the mail facility.
• *.emerg * broadcasts emergency messages from all facilities to logged-in terminal users.
• @host conventionally forwards over UDP, while @@host conventionally forwards over TCP.
• Rules can overlap, so one message can have multiple destinations.
• Configuration changes require a reload or restart and should be verified with a controlled logger test.

For related study, see Rsyslog.