VMware ESXi and vSphere Cluster Management

Configure a Splunk Universal Forwarder on Linux

Learn how to configure an installed Splunk Universal Forwarder on Linux, enable boot-start, forward logs to an indexer, monitor syslog, and verify ingestion.

A Splunk Universal Forwarder is a lightweight agent installed on a Linux host to collect local data and send it to a Splunk indexer. This guide assumes the forwarder is already installed at /opt/splunkforwarder.

How Splunk forwarding works

A monitor input watches a file, directory, or file pattern for new data. The Universal Forwarder reads the data, adds metadata such as host, source, and sourcetype, and sends events to a configured forward server.

The forward server is usually a Splunk indexer. An indexer receives, processes, stores, and makes events searchable. The basic flow is:

1. A Linux log file receives new events.
2. The Universal Forwarder tails the monitored file.
3. The forwarder sends the events over the network to the indexer's receiving port.
4. The indexer stores the events in the selected index.
5. An administrator searches the indexed events in Splunk.

Do not confuse the two important network roles:

• Management port: Used by the Splunk CLI and administrative operations. The local Splunk management service commonly uses port 8089.
• Receiving port: Used by an indexer to accept forwarded data. TCP port 9997 is a common receiving-port choice.

The destination index must already exist, and the indexer must have a receiving input enabled. Creating a forwarder destination does not create the index or enable the indexer's receiver.

Prerequisites

Before changing the forwarder configuration, collect the following information and confirm that the required permissions and network access are available.

• Have shell access with sufficient privileges to configure the service and read the selected log files.
• Confirm that the indexer receiver is enabled, commonly on TCP port 9997.
• Test DNS resolution and TCP connectivity from the Linux host to the indexer receiving port.
• Confirm that the target index exists and that the forwarder is allowed to write to it.
• Know the current Splunk administrative credentials. Do not assume that default credentials remain valid.

Run the Splunk CLI from the forwarder installation

The standard Splunk command-line utility is located in /opt/splunkforwarder/bin. You can change to that directory:

cd /opt/splunkforwarder/bin

Alternatively, invoke the executable by its full path:

/opt/splunkforwarder/bin/splunk status

Run commands as the user that owns and operates the forwarder when possible. Use sudo when the operation requires system-level service registration, access to protected files, or ownership and permission changes. Avoid running some commands as root and others as an unprivileged account if that creates inconsistent file ownership.

On first startup, Splunk may require license acceptance and creation or confirmation of an administrative account. Complete that initialization before using authenticated management commands.

Enable automatic startup at boot

boot-start is Splunk's service integration that configures the Universal Forwarder to start when Linux boots. From the forwarder command directory, run:

cd /opt/splunkforwarder/bin
sudo ./splunk enable boot-start

The command normally creates or registers an operating-system service. On systemd-based distributions, this is commonly represented by a systemd unit, although the exact unit name and generated service details can vary by Splunk version and installation choices.

Check the forwarder immediately after enabling boot-start:

sudo ./splunk status

On a systemd host, also inspect the registered service with the service name shown by the boot-start command or your operating-system configuration:

systemctl status <forwarder-service-name>
systemctl is-enabled <forwarder-service-name>

For a full validation, reboot during an approved maintenance window and verify that the service is running and enabled afterward. Review the service manager and forwarder logs if it does not start.

Configure the indexer as a forwarding destination

Add the indexer hostname and receiving port as a forward server. Replace the placeholders with environment-specific values:

cd /opt/splunkforwarder/bin
./splunk add forward-server <INDEXER_HOST>:9997 -auth <USERNAME>:<PASSWORD>

For example, splunk-indexer.example.net:9997 identifies the indexer receiver. The credentials in this command authenticate the local management action; they do not provide network access to TCP port 9997. DNS, routing, firewalls, and the indexer's receiving configuration must still permit the connection.

List configured destinations and inspect their connection state:

./splunk list forward-server

A healthy destination should be listed and show an active or connected state. If it is disconnected, check the hostname, receiver port, firewall rules, DNS, TLS requirements, and indexer receiver status.

Configure a file-monitoring input

Use add monitor to tell the forwarder what data to tail and which metadata to assign:

sudo ./splunk add monitor <LOG_PATH> -sourcetype <SOURCE_TYPE> -index <INDEX_NAME>

Each option has a specific purpose:

• <LOG_PATH> identifies the file, directory, or supported file pattern to monitor.
• -sourcetype assigns a label describing the event format or category. Consistent sourcetypes improve parsing and searching.
• -index selects the logical Splunk data store for the events. The index must exist on the receiving Splunk environment.

The forwarder process must be able to read the selected file. Check file ownership, group membership, mode bits, ACLs, and any security policy such as SELinux. Use the least privilege necessary rather than making logs broadly readable.

Production configurations should account for rotation. A rotated file may be renamed, compressed, or replaced, so verify that the chosen file or pattern matches the logging implementation. Directory and wildcard monitoring can collect multiple files, but overly broad patterns may ingest unwanted data or duplicate events.

Worked example: monitor Linux syslog

The following example monitors Debian- or Ubuntu-style syslog, assigns the sourcetype linux_logs, and sends events to the remotelogs index:

cd /opt/splunkforwarder/bin
sudo ./splunk add monitor /var/log/syslog -sourcetype linux_logs -index remotelogs

These names are environment-defined values, not universal requirements. Match your organization's index and sourcetype naming conventions. Some RHEL- or CentOS-style systems write traditional syslog events to /var/log/messages instead:

sudo ./splunk add monitor /var/log/messages -sourcetype linux_logs -index remotelogs

Some modern systems use a systemd journal without writing the desired events to either file. In that case, identify the approved collection method for the journal rather than monitoring a nonexistent path.

Apply the configuration changes

Restart the forwarder after adding the destination or monitor input so the running process loads the updated configuration:

sudo ./splunk restart
sudo ./splunk status

Restarting is also a useful checkpoint: a configuration or permission error may be reported during the restart or appear in the forwarder logs. Do not rely only on a successful process start; verify the forwarding connection and event delivery separately.

Verify end-to-end event delivery

1. Verify the local process

sudo ./splunk status

Confirm that the Universal Forwarder is running. If it is stopped, inspect startup output and the forwarder logs under the installation's log directory.

2. Verify the forwarding connection

./splunk list forward-server

Confirm that the configured indexer is listed and connected. A running forwarder with a disconnected destination cannot deliver events.

3. Generate or wait for new data

Monitoring primarily concerns new data written after the input is configured. Wait for a normal syslog event or create an approved test event. Do not alter production logs merely to test ingestion.

4. Search the indexer

In the Splunk search interface or an indexer search context, start with a time-bounded search:

index=remotelogs sourcetype=linux_logs source=/var/log/syslog earliest=-15m latest=now

You can narrow the search by host:

index=remotelogs sourcetype=linux_logs host=<linux-hostname> earliest=-15m latest=now

The metadata values may differ from expectations if an existing configuration overrides them. If no results appear, search the index over a slightly broader time range and then add host, source, or sourcetype filters one at a time.

Event visibility can be delayed by network transmission, indexer ingestion, parsing, indexing, or search-time factors. Verify newly generated events rather than relying only on old log content.

Configuration persistence and inspection

Splunk stores input and output settings in configuration areas. The two important files for this task are:

• /opt/splunkforwarder/etc/system/local/inputs.conf — defines data inputs such as monitored files.
• /opt/splunkforwarder/etc/system/local/outputs.conf — defines forwarding destinations and output behavior.

CLI commands write or manage the underlying configuration. Inspect these files and the effective configuration when troubleshooting, but do not make unmanaged direct edits that conflict with CLI-generated settings, a Deployment Server, configuration-management software, or another centrally managed source.

A centrally managed deployment may overwrite local changes. Establish which configuration layer is authoritative before editing files manually, and keep index, source, host, and sourcetype naming consistent across hosts.

Common troubleshooting

Forwarder cannot connect to the indexer

• Confirm the configured hostname and receiver port with ./splunk list forward-server.
• Test DNS resolution and TCP reachability from the forwarder host to the receiver.
• Confirm that the indexer is listening for forwarded data and that its receiving input is enabled.
• Check firewalls, routing, and any TLS or certificate requirements.
• Review forwarder logs and the destination connection state.

Forwarder runs but no events appear

• Verify that the monitored path exists and matches the actual Linux logging setup.
• Confirm that new data is being written to the file.
• Check that the forwarder service account can read the file.
• Confirm that the destination index exists and is searchable.
• Search with a broader time range before filtering by host, source, or sourcetype.
• Inspect effective configuration for routing, filtering, or duplicate inputs.

Forwarder does not start after reboot

• Confirm that boot-start completed successfully.
• Check service-manager status and whether the service is enabled.
• Review startup logs and verify service-account permissions.
• Re-run boot-start configuration with the correct privileges if the registration is missing.

Monitor command fails or the file is unreadable

• Use ls to verify the path and spelling.
• Identify the active system logging configuration and select the correct file.
• Check ownership, permissions, ACLs, and security policy restrictions.
• Confirm that the sourcetype and index arguments are spelled correctly.
• Inspect the resulting input configuration.

CLI authentication fails

• Use the current administrative account rather than assuming default credentials.
• Confirm that the local Splunk management service is running.
• Check the management connection separately from the indexer receiving connection.
• Avoid placing reusable passwords in command history, scripts, or process listings.

Operational and security considerations

• Use least privilege while preserving access to the log files the forwarder must read.
• Use trusted network paths and configure TLS and certificate validation when required by policy.
• Do not collect sensitive files without an approved data-handling, access-control, and retention plan.
• Define consistent index, sourcetype, source, and host naming before deploying many forwarders.
• Prefer centrally managed configuration when the environment uses a Deployment Server or configuration-management platform.
• Protect administrative credentials with interactive authentication or an approved secret-management solution.

For a concise reference to this procedure, see Configure a Splunk Forwarder on Linux.