What Are Splunk Forwarders?
Learn what Splunk forwarders do, how universal forwarders collect remote logs, how inputs.conf and outputs.conf work, and how indexers receive and search forwarded events.
A Splunk forwarder is a Splunk component that reads machine data and sends it to another Splunk component for processing. It places data collection close to the source while allowing a central Splunk deployment to perform indexing, storage, and search.
Forwarders are the data-collection layer in a distributed Splunk deployment. For example, instead of installing a complete indexing system on every web server, you can install a lightweight forwarder on each server and send the collected events to one or more central indexers.
What Is a Universal Forwarder?
A universal forwarder is a lightweight, dedicated Splunk distribution designed primarily to collect and send data. It includes the components needed to read configured inputs and forward events to a receiving Splunk instance.
Universal forwarders are suitable for remote servers and endpoints because they use fewer resources and have a focused role. They can collect broad categories of input data, using approaches such as monitored files, directories, network inputs where supported, and Windows Event Log channels. Unlike a full Splunk Enterprise instance, however, a universal forwarder operates as a forwarding-only component.
The term input means a configured source that Splunk reads. Examples include a log file, a directory, a network source, or a Windows Event Log channel. The term forwarding means sending the collected event data to a receiving Splunk instance.
What Universal Forwarders Do Not Do
A universal forwarder does not provide the complete set of capabilities found in Splunk Enterprise. In particular, it does not:
- Index data locally.
- Provide Splunk Web, the browser-based Splunk user interface.
- Include the bundled Python executable and libraries found in a full Splunk Enterprise installation.
- Provide local indexing and search availability for the events it collects.
Indexing and search availability occur on the receiving indexer. The forwarder reads and transmits events; the indexer performs the central processing and storage work.
| Capability | Universal Forwarder | Indexer or Full Splunk Enterprise Instance |
|---|---|---|
| Collects input data | Yes | Yes |
| Forwards data | Yes | Can be configured to do so |
| Indexes data locally | No | Yes for an indexer |
| Provides Splunk Web | No | Available on a full Splunk Enterprise installation when configured |
| Includes bundled Python components | No | Included with a full Splunk Enterprise installation |
| Primary deployment role | Remote collection and forwarding | Indexing, storage, search, and broader Splunk functions |
Forwarder-to-Indexer Data Flow
A typical forwarding path has these stages:
- A source host generates data, such as a web server writing access and error logs.
- The forwarder reads the configured input on that host.
- The forwarder sends the collected events across the network.
- A receiving Splunk instance accepts the connection through a listening receiver endpoint.
- The indexer processes and stores the received events.
- Users search the indexed events from the central Splunk deployment.
| Stage | Component | Responsibility |
|---|---|---|
| Data generation | Source host or application | Creates files, operating-system events, or other machine data |
| Input consumption | Forwarder | Reads data from the configured local input |
| Network forwarding | Forwarder | Sends events to the configured receiving endpoint |
| Receiving | Receiver | Accepts forwarded data on a listening host and port |
| Indexing and storage | Indexer | Processes events and stores indexed data |
| Search | Central Splunk deployment | Makes indexed events available to users and searches |
A receiver is a configured Splunk instance and listening port that accepts data sent by forwarders. The receiver must be enabled on the intended receiving instance, and its host and port must match the destination configured on the forwarder.
Input configuration and forwarding-destination configuration are separate responsibilities. The forwarder needs to know both what to read and where to send it.
Example: Centralize Web-Server Logs
Consider a fleet of web servers. Each server produces local access logs, error logs, and system logs. A common design is:
- Install a universal forwarder on every web server.
- Configure file-monitoring inputs for the local web access and error logs.
- Configure each forwarder to send events to a central indexer receiver.
- Let the indexer process and store the events.
- Search all web-server events centrally instead of logging in to each server.
This design separates collection from indexing. The web servers run a focused collection component, while central indexers handle the storage and search workload.
For installation and platform-specific configuration, see Install a Splunk Forwarder on Linux, Install a Splunk Forwarder on Windows, and Configure a Splunk Forwarder on Linux.
Configuring Data Inputs with inputs.conf
inputs.conf is the primary configuration file used to define what data a forwarder consumes. Input stanzas identify sources such as monitored files, monitored directories, or Windows Event Log channels.
Common metadata choices include:
- host: Identifies the machine that generated the event.
- source: Identifies the specific input source, such as a file path.
- sourcetype: Describes the format or kind of event data.
- index: Specifies the destination index when the deployment permits the input to assign one.
Deploy an input configuration on the forwarder that can access the local data source. A forwarder on one host cannot monitor a file or event channel that exists only on another host unless a suitable remote collection method is configured.
Monitor a Local Log File
[monitor:///var/log/example/application.log]
disabled = false
sourcetype = example:application
index = main
- Use a path that exists on the forwarder host.
- Make sure the forwarder service account can read the file.
- Choose a sourcetype and destination index appropriate for the deployment.
For more input examples, see Monitor Logs Using Forwarders, Data Sources, and Add Data to Splunk.
Configuring Forwarding Destinations with outputs.conf
outputs.conf specifies one or more receiving indexers or receiver endpoints. A forwarding target typically includes a hostname and a TCP port.
[tcpout]
defaultGroup = primary_indexers
[tcpout:primary_indexers]
server = indexer.example.com:9997
Port 9997 is a commonly used Splunk-to-Splunk receiving port. A deployment may use a different approved port, so the value in outputs.conf must match the port enabled on the receiver.
Basic destination validation includes:
- Confirm that the forwarder can resolve and reach the receiver hostname.
- Confirm that the receiving Splunk instance is available.
- Confirm that the receiver is listening on the configured port.
- Check host and network firewalls for blocked connections.
- Verify that events arrive at the expected index after forwarding.
For receiver configuration, see Set Up a Receiver.
Setting Up a Receiver
The receiving indexer, or another intended receiving Splunk instance, must be configured to accept forwarded data. Enable a receiving port on that instance and use the same host and port in the forwarder's output settings.
[splunktcp://9997]
disabled = false
Apply this receiver configuration to the receiving Splunk Enterprise instance, typically an indexer. Ensure that network and host firewall rules permit connections from the forwarder hosts.
The receiver configuration and the forwarder's outputs.conf must agree. For example, if the receiver listens on port 9997, the forwarder must target that receiver host on port 9997, unless the deployment intentionally uses another port.
Collecting Windows Event Logs
A universal forwarder installed on Windows can collect Windows Event Log channels. Common channels include Application, System, and Security. The forwarder reads those events and sends them to the indexer for centralized indexing and search.
[WinEventLog://Application]
disabled = false
[WinEventLog://System]
disabled = false
[WinEventLog://Security]
disabled = false
The account running the forwarder must have permission to read each selected channel. Security events may require elevated or specifically assigned permissions. Confirm that the expected events exist locally in Windows Event Viewer.
See Windows Inputs, Collect Event Logs From a Local Windows Machine, and Collect Windows Host Information for related Windows collection tasks.
Common Forwarder Input Types
| Input Type | Typical Source | Example Use |
|---|---|---|
| Monitored file | Application or web log file | Collect an access log or error log as it grows |
| Monitored directory | Directory containing rotating logs | Collect multiple files from one application log directory |
| Windows Event Log | Application, System, Security, or custom channel | Collect operating-system and security telemetry |
| Network input where supported | Network source | Receive supported data directly from a network producer |
Configuration Responsibilities
| Configuration | Primary Question | Typical Location |
|---|---|---|
inputs.conf | What data should this component read? | Forwarder that can access the source |
outputs.conf | Which receiver or indexer should receive the data? | Forwarder sending the events |
| Receiver input configuration | Which port accepts forwarded data? | Receiving Splunk instance |
Troubleshooting Forwarding
No Events Appear in Search
If a forwarder is installed but no events appear in Splunk search, check the complete path from source to indexer:
- Confirm that the monitored file exists or that the expected event-log channel is present.
- Confirm that the input stanza is enabled and has valid syntax.
- Confirm that the forwarder service account can read the source.
- Verify the receiver hostname, port, DNS resolution, and network reachability.
- Confirm that the receiving port is enabled on the indexer.
- Check firewalls between the forwarder and receiver.
- Verify that events arrive in the intended index and use the expected sourcetype.
Windows Event Data Is Missing
If file inputs work but Windows Event Log data is missing:
- Confirm that the required Application, System, Security, or custom channel stanza exists and is enabled.
- Verify the forwarder service account's permissions, especially for Security events.
- Check the channel name and confirm that events exist in Windows Event Viewer.
Events Go to the Wrong Destination or Index
- Inspect
outputs.conffor the correct receiver group, hostname, and port. - Inspect the input metadata for an unintended
indexvalue. - Check active configuration and configuration precedence for an overriding setting.
Forwarders and Indexers: The Architectural Boundary
The most important distinction is that a universal forwarder performs collection and transmission, while an indexer performs indexing and central storage. This boundary makes it practical to collect data from many remote systems without running a full indexing installation on every host.
After the indexer receives and processes the events, users can search them through the central Splunk interface. A forwarder itself is not the place to open Splunk Web or search locally indexed data because it does not provide Splunk Web or local indexing.
Exam-Relevant Notes
- A forwarder consumes data and sends it to another Splunk component.
- A universal forwarder is a lightweight distribution focused on remote collection and forwarding.
inputs.confdefines what the forwarder reads.outputs.confdefines where the forwarder sends data.- The receiving indexer must have a listening receiver endpoint.
- Universal forwarders do not locally index data, provide Splunk Web, or include the bundled Python components of full Splunk Enterprise.
- Windows Event Log collection requires access to the selected channels.
Next Steps
Once the collection path is working, continue with What Is an Index? to understand where events are stored, or review Monitor Remote Windows Event Logs for a Windows-focused collection scenario.