VMware ESXi and vSphere Cluster Management

What Are Splunk Forwarders?

Learn how Splunk forwarders collect logs and host data, send it to indexers, and fit into centralized Splunk deployments.

A Splunk forwarder is a Splunk component installed near a data source. It collects machine data from that host and sends the data to another Splunk component, usually an indexer.

Forwarders are commonly installed on servers, workstations, application hosts, web servers, and other systems that generate logs, metrics, and event data. Instead of manually copying files to a central server, each host can collect its own data and transmit new events as they are created.

This approach is more efficient and manageable because collection happens close to the source. It reduces manual file transfers, provides a consistent configuration method, and allows administrators to manage data from many systems in a central Splunk environment.

What Is a Universal Forwarder?

The universal forwarder is Splunk's lightweight, dedicated forwarding agent. It is designed for efficient deployment across many remote servers and endpoints.

A universal forwarder includes the functionality needed to define data inputs, read or receive local data, and forward that data to a receiving Splunk component. Its reduced footprint makes it suitable for widespread installation.

Unlike a full Splunk Enterprise instance, a universal forwarder does not provide the complete Splunk platform. In particular, it has:

• No Splunk Web browser interface.
• No bundled Python runtime and libraries used by the full platform.
• No local indexing function for creating searchable indexes.

A full Splunk Enterprise deployment can collect data, index it, store it, and provide search and administration features. A universal forwarder focuses on collection and transmission instead.

Forwarders, Indexers, and Search

Several separate activities are involved in getting machine data into a Splunk search:

1. Data input: Splunk reads or receives data from a configured source, such as a file, directory, network port, or Windows Event Log channel.
2. Forwarding: The forwarder transmits collected data to a configured forwarding destination.
3. Receiving: An indexer or another configured Splunk destination accepts the forwarded connection on a receiving port.
4. Indexing: The indexer processes incoming events and organizes them into searchable indexes.
5. Storage: The indexer stores the indexed data according to retention and storage settings.
6. Search: Users search the centrally stored data through the Splunk search environment.

The forwarder does not create local searchable indexes for the data it collects. The receiving indexer performs the processing and indexing work.

Typical Forwarding Architecture

The common deployment pattern is many to one: multiple source hosts run forwarders and send data to one or more indexers.

A web server might run a universal forwarder that monitors access and error logs. A Windows server might run a forwarder that collects Application, System, Security, and PowerShell-related events. An application host might monitor its application log directory. All of these forwarders can send data to a central indexer or an indexer group.

The indexer acts as the receiving destination. It accepts incoming forwarded data, indexes it, stores it, and makes it available to centralized searches. Centralized retention and search allow administrators to investigate many distributed systems without logging into each host separately.

In larger environments, a forwarder can be configured with multiple forwarding destinations. This supports scale and resilience by allowing traffic to be distributed among indexers and providing alternate destinations when appropriate.

How Forwarders Collect Data

A forwarder reads or receives data locally before transmitting it onward. The data source is defined as a data input, which is a configured source from which Splunk reads or receives data.

Configuration Fundamentals

Input definitions and forwarding destinations are separate configuration concerns. Data inputs are typically defined in inputs.conf. Forwarding targets and forwarding behavior are commonly defined in outputs.conf.

Monitoring a Web Log File

This input tells a forwarder to monitor a local web access log, send the events to the web index, and identify them with the access_combined source type:

[monitor:///var/log/web/access.log]
disabled = 0
index = web
sourcetype = access_combined

The path must exist on the forwarder, and the forwarder service must have permission to read it. The target index must also exist or be accepted according to the receiving environment's index configuration.

Configuring Forwarding Destinations

The forwarder's outputs.conf identifies the receiving host and port. In this example, the forwarder sends data to an indexer at port 9997:

[tcpout]
defaultGroup = primary_indexers

[tcpout:primary_indexers]
server = indexer.example.com:9997

The forwarding destination is the target host or group of hosts to which collected data is sent. In production, the group may contain multiple indexers rather than a single server.

Enabling the Indexer Receiver

The receiving indexer must be configured to listen for forwarded data. A typical receiving input uses the same port named in the forwarder's outputs.conf:

[splunktcp://9997]
disabled = 0

The forwarder and receiver must have network connectivity, compatible host and port settings, and any required firewall rules. A configured destination alone is not enough if the indexer is not listening.

Applying Configuration Changes

After changing configuration, validate the files and settings. Some changes can be applied through supported configuration reloads, while others require a service restart. When a restart is applicable, use the appropriate command for the installed operating system and deployment method:

splunk restart

Always check the effective configuration and service logs after making changes. Configuration files can be supplied by deployment tools or applications, and a higher-precedence setting can override a local file.

Windows Host Data Collection

A universal forwarder on Windows can collect Windows Event Log channels and other Windows host information. Common channels include:

• Application: events generated by applications and services.
• System: operating system, driver, and service events.
• Security: authentication, authorization, audit, and security-policy events.
• PowerShell-related events: events useful for investigating PowerShell activity, when the relevant logging is enabled.

Example input definitions for common channels are:

[WinEventLog://Application]
disabled = 0

[WinEventLog://System]
disabled = 0

[WinEventLog://Security]
disabled = 0

The forwarder service account must be permitted to read every selected channel. Security logs commonly require particular permissions, service-account settings, or local policy configuration. If Application and System events arrive but Security events do not, check permissions and confirm that the Security input is enabled.

End-to-End Data Flow

Practical Example: Centralized Web Server Logs

1. Install a universal forwarder on each web server.
2. Configure each forwarder to monitor its access and error log files.
3. Assign an appropriate source type and target index.
4. Configure the forwarders to send data to a central indexer receiver.
5. Search web activity centrally rather than logging into each web server.

This design also makes correlation easier. A web request error can be compared with application events, operating system messages, and events from other servers in one search environment.

Practical Example: Remote Application Logs

1. Configure a forwarder to monitor the application's log directory.
2. Set an appropriate source type and target index in the input definition.
3. Allow the forwarder to read new events as the application writes them.
4. Forward those events to an indexer.
5. Use centralized searches to correlate application failures with infrastructure events.

Operational Validation

Validation should cover the forwarder, the network connection, the receiver, and the search result. A forwarder can be running while still failing to read a file or connect to its destination.

1. Check the service: Confirm that the forwarder process is running on the source host.
2. Check inputs: Verify the file path, directory, port, or Windows channel in inputs.conf.
3. Check permissions: Confirm that the service account can read the source data.
4. Check output status: Run splunk list forward-server and review connection state.
5. Check forwarder logs: Look for input, permission, DNS, connection, or queue errors.
6. Check the receiver: Confirm that the indexer is listening on the configured receiving port.
7. Check the network: Verify DNS resolution, routing, firewall rules, and port reachability.
8. Check search results: Search the expected index over a sufficiently broad time range.

For example, a search by the forwarder host can help confirm arrival:

index=web host=<forwarder_host>

If the index or source type is uncertain, begin with a broader search by host or source. Once events are found, inspect their actual index, source, and source type before narrowing the search.

Troubleshooting Common Problems

No Events Appear in Search

• Confirm that the monitored file path or Windows input is correct.
• Check that the source is producing new readable events.
• Verify that the forwarder service is running.
• Review the target index and expand the search time range.
• Inspect forwarder logs for input or permission errors.

The Forwarder Cannot Connect to the Indexer

• Confirm that the indexer receiver is enabled.
• Compare the host name and port in outputs.conf with the receiver configuration.
• Check firewall and network rules between the hosts.
• Test DNS resolution and network reachability.
• Use splunk list forward-server and forwarder logs to identify connection failures.

Windows Security Events Are Missing

• Confirm that [WinEventLog://Security] is enabled.
• Verify that the forwarder service account can read the Security channel.
• Review relevant Windows policies and event filtering.
• Use Windows Event Viewer to confirm that the expected events exist locally.

Data Arrives in an Unexpected Index or Source Type

• Review the input stanza's index and sourcetype settings.
• Search broadly by host or source before applying restrictive criteria.
• Check whether another application or deployment configuration overrides the local file.
• Review configuration precedence and the effective input configuration.

Exam-Relevant Distinctions

• A forwarder is installed near the data source and is responsible primarily for collection and transmission.
• A universal forwarder is lightweight and does not include Splunk Web, a bundled Python runtime and libraries, or local indexing.
• An indexer receives data, indexes it, stores it, and supports searches against that data.
• inputs.conf defines what the forwarder collects.
• outputs.conf defines where the forwarder sends collected data.
• The indexer receiver must listen on a port that matches the forwarder's configured destination.
• Missing data can result from an incorrect input, missing permissions, a stopped service, a blocked network path, an unavailable receiver, or an overly narrow search.

Summary

Splunk forwarders provide distributed data collection for centralized Splunk deployments. A universal forwarder runs on remote systems, reads local files, directories, network inputs, or Windows Event Log channels, and sends the collected data to an indexer. The indexer receives and indexes the events, stores them according to retention settings, and makes them searchable from the central environment.

The essential configuration relationship is straightforward: define data sources in inputs.conf, define forwarding destinations in outputs.conf, enable a matching receiver on the indexer, and validate the service, network path, incoming data, and search results.