Splunk online course

Configure a Splunk Universal Forwarder on Linux

Configure a Splunk Universal Forwarder on Linux to start at boot, send log data to an indexer, monitor files, and verify indexed events.

What the Universal Forwarder Does

The Splunk Universal Forwarder is a lightweight Splunk component that collects machine data and sends it to a Splunk indexer. It does not normally provide the full search and indexing experience on the source host.

The forwarder host is the Linux machine where the logs exist and where the Universal Forwarder runs. The indexer host is the Splunk server that receives, processes, stores, and makes the events searchable.

The typical data flow is:

1. A Linux application or operating-system service writes to a log file.
2. A monitor input on the Universal Forwarder reads newly appended log content.
3. The forwarder sends events over TCP to a configured forward-server, usually an indexer receiving port.
4. The indexer assigns metadata, stores the events in an index, and makes them available to Splunk Search.

A receiving port is a network port on an indexer that accepts forwarded Splunk data. 9997 is a common receiving-port choice, but the actual port must match the indexer's configuration.

Prerequisites

Complete these checks before changing the forwarder:

• Confirm that the Universal Forwarder is installed. The commonly used installation directory is /opt/splunkforwarder.
• Obtain sudo or equivalent administrative privileges.
• Identify the indexer hostname or IP address and its receiving port.
• Confirm that the forwarder can resolve and reach the indexer through DNS, routing, and firewall policy.
• Confirm that the indexer is configured to receive forwarded data on the intended port, commonly 9997.
• Confirm that the destination index exists and that the relevant Splunk role or receiving configuration permits data to be written to it.
• Identify the actual Linux log path. Do not assume that every distribution uses the same file.

Values to Collect Before Setup

Find the Forwarder Command Directory

Run the Splunk CLI from the Universal Forwarder's bin directory. The default installation path is commonly:

cd /opt/splunkforwarder/bin

If the forwarder was installed elsewhere, replace the path with the actual installation directory. Confirm that the executable is present before continuing.

Enable Automatic Startup

boot-start configures the forwarder to start automatically with the operating system. Run:

sudo ./splunk enable boot-start

The exact service-manager integration depends on the Linux distribution and whether it uses systemd, another init system, or a customized service configuration. Follow the command output and verify the resulting operating-system service configuration.

Check the forwarder immediately:

sudo ./splunk status

After a test reboot, run the same status command and also inspect the operating system's service-manager status and boot logs if the process did not start.

Configure the Indexer Destination

Add a forward-server target using the format HOST:PORT:

./splunk add forward-server <indexer-host>:9997 -auth <admin-user>:<admin-password>

For example, an environment might use:

./splunk add forward-server splunk-indexer.example.net:9997 -auth <admin-user>:<admin-password>

Replace every placeholder with an environment-specific value. Splunk CLI configuration actions require authentication with an account authorized to administer the forwarder.

List the configured destinations:

./splunk list forward-server

The intended indexer hostname or IP and receiving port should appear in the output. Prefer DNS names or stable addressing when appropriate, so an address change does not require editing every forwarder.

Add a Monitored Linux Log Input

A monitor input watches a file, directory, or path pattern and reads newly appended content as events. Add a file monitor with a sourcetype and destination index:

sudo ./splunk add monitor <log-path> -sourcetype <sourcetype-name> -index <index-name>

For a Debian- or Ubuntu-style system where the file exists, an example is:

sudo ./splunk add monitor /var/log/syslog -sourcetype linux_logs -index remotelogs

The sourcetype is metadata that identifies an event format and supports parsing, field extraction, and search behavior. The index is the named Splunk data store used for event organization, permissions, and retention.

A monitor can target one file, a directory, or a pattern-based path. Use narrow paths when possible. A broad directory or overlapping monitors can collect unintended files or create duplicate data.

The forwarder's service account must be able to read the selected file and traverse each parent directory. Check Linux ownership and permissions if collection fails. Also verify that the path is a text-file log source; some systems use systemd journal storage rather than a traditional syslog file.

Common Linux Log Path Differences

Apply and Validate the Configuration

Restart the forwarder after changing its forwarding or input configuration:

sudo ./splunk restart

Then verify that it is running:

sudo ./splunk status

Check the destination again:

./splunk list forward-server

In Splunk Search, begin with a constrained query using the destination index, forwarder host, and sourcetype:

index=<index-name> host=<forwarder-host> sourcetype=<sourcetype-name>

For the example configuration, use equivalent values such as index=remotelogs host=<forwarder-host> sourcetype=linux_logs. Select a time range that includes the expected event time.

If the source file is quiet, wait for a new event or generate a harmless event through a normal operating-system or application action. Searching for an event that was written before the monitor was configured may not prove that the new input is working.

Configuration Persistence and Management

Command-line additions are persisted in Splunk configuration. The main conceptual configuration areas are:

• outputs.conf: Defines forwarding destinations and related forwarding behavior.
• inputs.conf: Defines data inputs, including monitored files and directories.

Review the effective configuration when troubleshooting. A pre-existing app, local configuration, or deployment tool can override or supplement settings created by a CLI command.

For production, manage forwarder settings through approved deployment tools or configuration management where practical. This improves consistency, change tracking, and repeatability instead of relying on undocumented ad hoc commands.

Operational and Security Considerations

• Set a non-default administrative password and protect all Splunk credentials.
• Restrict access to Splunk management interfaces and administrative ports.
• Use stable DNS names or addressing for indexer targets where appropriate.
• Choose consistent names for source, sourcetype, host, and index. These values affect searches, parsing, ownership, permissions, and retention planning.
• Obtain approval before indexing sensitive files. Apply appropriate filtering and access controls, and avoid collecting secrets or unnecessary personal data.
• Ensure firewall rules permit only the required forwarder-to-indexer traffic.

Troubleshooting

No Events Appear in the Expected Index

• Confirm that the forwarder was restarted after the configuration change.
• Run sudo ./splunk status and ./splunk list forward-server.
• Check the monitor path, sourcetype, and index settings.
• Search more broadly by host or source before narrowing the query.
• Confirm that the selected file exists and is receiving new content.
• Verify that the destination index exists and that the receiving configuration permits writes to it.

The Forwarder Cannot Connect to the Indexer

• Check the hostname, IP address, and port for typographical errors.
• Verify DNS resolution and network reachability from the forwarder.
• Confirm that firewall policy allows the receiving port.
• Confirm that the indexer is listening on the configured receiving port.
• Review forwarder and Splunk logs for connection failures.

The Forwarder Starts Manually but Not After Reboot

• Confirm that enable boot-start completed successfully.
• Inspect the Linux service manager and boot logs.
• Check the configured startup user, installation ownership, and permissions.
• Account for differences between systemd and other init systems.

The Monitored Log Is Not Collected

• Verify the path and file permissions.
• Confirm that the forwarder service account can read the file and traverse its directories.
• Determine whether the system uses a journal-only setup rather than a text log.
• Generate or wait for a new line of log data.
• Review the active input configuration and forwarder logs.

Events Arrive with Unexpected Metadata

• Search by source to locate the events.
• Check the effective inputs.conf settings.
• Confirm the spelling of the index and sourcetype.
• Check whether another input or configuration layer overrides the intended values.
• Verify the host value reported by the forwarder and the indexer.

Configuration Sequence

Next Steps

For related configuration work, see Install a Splunk Forwarder on Linux, Set Up a Receiver, Monitor Logs Using Forwarders, Create an Index, and What Are Forwarders.