IT Course Directory: VMware, Linux, Networking, and Raspberry Pi

Splunk Fundamentals: Installation, Data Collection, Search, Forwarding, Alerts, and Reports

Learn Splunk Enterprise fundamentals: install it on Windows or Linux, ingest logs, search with SPL, collect Windows data, configure forwarders, and create alerts and reports.

Splunk Enterprise is a platform for collecting, indexing, searching, monitoring, and analyzing machine-generated data. This lesson is designed for beginners in IT operations, system administration, security monitoring, and data analysis.

You will install or access Splunk, add local and forwarded data, write Search Processing Language (SPL) searches, collect Windows events and performance data, configure universal forwarders, and save searches as alerts and reports.

Splunk platform overview

Splunk Enterprise is the deployment used to ingest, index, search, and manage machine data. Machine data includes text logs, application records, web-server logs, Windows Event Logs, performance measurements, and host inventory information.

Splunk turns incoming records into searchable events. An event is an individual timestamped record of machine data. Events are stored in named indexes, searched with SPL, and displayed as raw events, tables, charts, or other visualizations.

Common operational uses include application-log investigation, infrastructure monitoring, service health checks, Windows event analysis, performance monitoring, and security investigations such as failed authentication searches.

Installing and accessing Splunk

Windows installation

1. Obtain the Splunk Enterprise installer through your organization’s approved software process.
2. Run the Windows installer with an account permitted to install services and write to the selected installation directory.
3. Choose the installation location and configure the initial administrator account when prompted.
4. Review the license and initial setup screens. A lab may use an evaluation or other available license; production use requires the appropriate licensing arrangement.
5. Allow the installer to create and start the Splunk service if that option is presented.

After installation, confirm that the Splunk service is running in Windows service management. If it is stopped, start it using the service controls or the supported Splunk administration method for your installation.

Linux installation

1. Install the package or archive supplied through your approved Splunk software process.
2. Place the installation in a controlled directory and identify it as $SPLUNK_HOME.
3. Start Splunk and accept the license interactively:

$SPLUNK_HOME/bin/splunk start --accept-license

Create or confirm the local administrative account during first startup. To configure startup behavior on systems where this is appropriate:

$SPLUNK_HOME/bin/splunk enable boot-start

Use the operating system’s service manager or Splunk’s supported command-line controls to confirm that the process remains running after startup. Protect the installation directory and administrative credentials.

Opening Splunk Web

Splunk Web normally listens on its management web port. Open the Splunk host name or address with that port in a browser, then sign in with the local administrative account. Do not assume that a remote browser can reach the port: host firewalls, network firewalls, and port conflicts can prevent access.

For a first-use check, confirm that the instance is running, sign in successfully, open the Search and Reporting app, and verify that the interface can execute a search. Also review the license or initial setup status before onboarding substantial data.

Splunk Web and first-use navigation

Splunk Home shows installed apps and available areas. Search and Reporting is the main workspace for ad hoc searches, saved searches, reports, alerts, and visualizations. Settings and data-input areas are used for indexes, inputs, users, apps, and other configuration.

In Search and Reporting, the main workflow is:

1. Select a useful time range with the time range picker.
2. Enter an SPL search.
3. Run the search and inspect the event, statistics, and visualization views.
4. Use the fields sidebar to discover fields found in the returned events.
5. Save a tested search as a report, alert, or other knowledge object when appropriate.

Raw events are the original searchable records. Extracted fields are named values such as host, status, or user. A statistics table is a result set produced by a command such as stats; it is not the same thing as the original event data. Visualizations represent the resulting data graphically.

Data sources, events, and indexes

Typical data sources include text files, application logs, web-server logs, Windows Event Logs, metrics or performance counters, and host metadata. A data input defines how Splunk obtains a source and how its events are classified.

An index is a named logical repository for stored events. Separate indexes can support different teams, environments, retention periods, access requirements, or data types. Index selection affects search scope, performance, permissions, and governance.

Create indexes with meaningful names that describe their purpose, such as training_app or windows. Before creating one, estimate event volume, storage size, retention requirements, and who should search it. Retention and size settings must fit the available storage and organizational policy; do not treat default values as a capacity plan.

Adding data to Splunk

Common ingestion paths are uploading a file for a short test, monitoring a local file or directory, enabling Windows inputs, and receiving data from forwarders.

1. Choose the source, such as a file, directory, Windows channel, or forwarded stream.
2. Select or confirm the sourcetype, which tells Splunk how the data is formatted.
3. Set host metadata so events can be attributed to the correct machine.
4. Choose the destination index.
5. Review timestamp recognition, event breaks, source, and sourcetype.
6. Submit the input, then verify that events are searchable.

Correct timestamp recognition is essential. If Splunk assigns the wrong event time, a search may appear empty even though data was ingested. Sourcetype assignment is also important because parsing, field extraction, and later searches often depend on it.

Start verification broadly, for example with the selected index and a wide time range. Then narrow by host, source, and sourcetype. Confirm that the event count changes when the source produces new data.

Search fundamentals and SPL syntax

SPL is Splunk’s Search Processing Language. A search begins with terms or predicates and can pass results through additional commands using the pipe, the vertical bar character.

index=training_app

Search terms can match words in events. Field-value predicates target extracted or default fields:

index=training_app level=ERROR host=app01

Use quoted phrases when a value contains spaces or punctuation:

index=training_app message="connection refused"

Wildcards can match variable text where supported, such as host=web*. Quote values containing spaces. Use the field name, an equals sign, and the value; escape or quote characters that would otherwise be interpreted as syntax.

Boolean expressions combine conditions. AND requires both conditions, OR requires either condition, and NOT excludes a condition. Parentheses make the intended grouping explicit:

index=training_app (level=ERROR OR level=WARN) NOT host=dev-test

Use the most specific appropriate index in the base search. A narrow base search usually improves speed, reduces irrelevant results, and makes conclusions easier to defend. The time range is equally important: it controls the events considered, affects performance, and can change the apparent severity of a problem. Select a period that matches the question, such as the last 15 minutes for an alert or the previous 24 hours for a daily report.

SPL uses a left-to-right pipeline. Each pipe sends the current result set to the next stage:

index=training_app | top limit=10 host | sort - count

The first stage finds events, top summarizes frequent hosts, and sort orders the resulting table. Later commands operate on the output produced by earlier commands.

Fields and interpreting results

Default fields are available for many events, including _time, host, source, sourcetype, and index. Extracted fields are values Splunk identifies from event content or that a search command creates. Field discovery in the results sidebar helps you learn which fields are present and how frequently they occur.

Fields can filter events, group results, calculate summaries, and define alert conditions. For example, host can compare systems, user can identify an account, and a calculated field such as event_count can be compared with a threshold.

Core SPL filtering and transforming commands

top is useful for prioritizing repeated values:

index=training_app level=ERROR | top limit=10 message

stats supports functions such as count, dc(field) for distinct count, avg(field), min(field), and max(field). Add by to group calculations:

index=training_app | stats count as event_count dc(user) as users avg(response_ms) as average_ms by host

A minus sign requests descending order; without it, sorting is ascending. Sort by multiple fields when needed:

| sort host, - event_count

Use base-search predicates when filtering raw events, because they reduce the input before later processing:

index=training_app level=ERROR host=app01

Use where when the condition depends on a field created or calculated by a previous command, especially after stats:

index=training_app | stats count as event_count by host | where event_count > 100 | sort - event_count

In a multi-stage search, always ask what the current result represents. After the base search it is events; after stats it is usually one row per group; after where it is a filtered table.

Windows data collection

Windows inputs collect local Windows Event Logs, performance counters, and host or operating-system inventory. Common Event Log channels include Application, System, and Security. Security data may require suitable service-account permissions and organizational approval.

In Splunk Web, select the Windows input type, choose channels or counters, assign the input to a Windows-focused index, review permissions, and submit. Validate with searches such as:

index=windows host=WIN01 sourcetype="WinEventLog:Security" (EventCode=4624 OR EventCode=4625)

index=windows host=WIN01 (EventCode=7036 OR EventCode=7040 OR EventCode=7045)

index=windows host=WIN01 (level=ERROR OR Type=Error)

Performance and inventory sourcetypes vary by input and deployment. Begin by searching the Windows host and inspecting available sourcetypes and fields before writing a narrow query. Confirm collection in Windows Event Viewer and then confirm the same activity in Splunk.

Universal forwarders and distributed collection

A universal forwarder is a lightweight Splunk agent that watches local data and sends it to a receiver. It is not a full Splunk Enterprise search and indexing instance. The sender-receiver model separates collection from central storage and analysis.

On the destination Splunk Enterprise instance, enable a receiving port and permit the forwarder network path:

[splunktcp://9997]
connection_host = dns

On a Linux forwarder, outputs.conf identifies the destination:

[tcpout]
defaultGroup = primary_indexer

[tcpout:primary_indexer]
server = splunk-receiver.example.net:9997

inputs.conf defines a monitored Linux log and its destination index:

[monitor:///var/log/myapp/app.log]
disabled = 0
index = training_app
sourcetype = myapp:log

After installing the Linux universal forwarder, start it, apply the configuration, and restart it after changes:

$SPLUNK_HOME/bin/splunk start --accept-license
$SPLUNK_HOME/bin/splunk restart

On Windows, install the universal forwarder with an approved installer, configure the receiving host and port, select local files or Windows inputs, assign indexes, and restart the forwarder service. A Windows Event Log stanza can look like this:

[WinEventLog://Application]
disabled = 0
index = windows

Validate a forwarder in three places: confirm its input is enabled and readable, confirm network connectivity to the receiver, and search the destination index for the expected host, source, and sourcetype. Remote Windows Event Log collection additionally requires suitable credentials, network access, enabled remote-event services where applicable, and permissions to read the requested channels.

Alerts

An alert is a saved search with a trigger condition and one or more actions. A scheduled alert runs at defined intervals and is usually appropriate for threshold checks over a recent time window. A real-time alert evaluates incoming events continuously and is useful when immediate response is more important than search efficiency.

1. Write and manually test the search with the intended time window.
2. Choose scheduled or real-time evaluation.
3. Define the trigger logic, such as a result count or a calculated threshold.
4. Set severity and suppression behavior appropriate to the operational impact.
5. Configure actions such as notification or an external script.
6. Generate a safe test condition and verify alert history and action delivery.

A result-count condition can trigger when a search returns one or more rows. A threshold condition compares a calculated field:

index=training_app level=ERROR earliest=-15m latest=now | stats count as error_count | where error_count > 50

External scripts require special care. Use trusted script paths, restrictive permissions, strict input validation, clear logging, and a dedicated test procedure. Do not pass unvalidated event content into shell commands. Confirm which account executes the action and test failure handling as well as success.

Reports and sharing

A report is a saved search intended for repeatable analysis, tabular output, or visualization. A saved search is the reusable object; it can become a report, alert, dashboard component, or scheduled task. An alert reacts to a condition, while a report communicates recurring findings.

1. Run and verify the search manually.
2. Save it with a clear title and description.
3. Set an appropriate time range or schedule.
4. Choose a table or visualization that supports the question.
5. Set sharing permissions deliberately.
6. Open the shared report using an appropriate test role and verify that its users can access the required indexes without exposing unnecessary sensitive fields.

Sharing scopes commonly include private, app-level, and broader scopes. Use the narrowest scope that meets the requirement. A report cannot reveal data that the viewing role cannot search, but its contents and fields should still be reviewed for privacy and security concerns.

Troubleshooting

Splunk Web does not open

Confirm that the Splunk service is running, verify network reachability to the management port, check for a port conflict, and review service logs for startup or binding errors. Confirm the browser uses the correct host, address, protocol, and port.

A monitored file produces no events

Check the path, permissions for the Splunk service account, and whether the input is enabled. Search with a broad time range and inspect by host or source before assuming the index or sourcetype is correct. Confirm that the file has received new content after monitoring began.

Windows events are missing

Confirm that the selected channel is enabled, the service account can read it, and the search uses the correct index. Search for the Windows host without assuming an event code, and compare activity with Windows Event Viewer.

A forwarder sends no data

Confirm the destination receiving port, firewall rules, host name, and port in outputs.conf. Check that an input in inputs.conf is enabled and readable, review forwarder internal logs, test connectivity, and restart after configuration changes.

Results show the wrong time period

Expand the time range, inspect raw event timestamps, review timestamp recognition for the sourcetype, and verify that the source system clock is synchronized.

An alert triggers too often or never triggers

Run the exact saved search manually for the scheduled period. Inspect the final result shape used by the trigger, verify that thresholds match normal volume, review alert history and suppression, and test notification or script actions independently.

End-to-end lab

1. Install Splunk Enterprise on a Windows or Linux lab host and confirm that Splunk Web and Search and Reporting work.
2. Create a training_app index with retention and size settings suitable for a small lab.
3. Upload or monitor an application log, assign a suitable sourcetype and host, and verify new events in the selected index.
4. Run index=training_app (ERROR OR FAIL) | sort - _time and inspect raw events and discovered fields.
5. Summarize volume with index=training_app | stats count as event_count by host | sort - event_count.
6. Filter noisy hosts with index=training_app | stats count as event_count by host | where event_count > 100 | sort - event_count.
7. If Windows data is available, enable Application or System events, assign the windows index, and validate by host and sourcetype.
8. If a second Linux host is available, enable a receiver, install a universal forwarder, configure inputs.conf and outputs.conf, restart it, and verify the remote host’s data.
9. Save the error-spike search as a scheduled alert, test its threshold and action, and inspect alert history.
10. Save a daily host summary as a report, choose a useful table or visualization, share it at the appropriate scope, and test access with the intended role.

Knowledge checks

1. What problem does an index solve, and how can index choice affect permissions and search performance?
2. Why should a search specify an appropriate time range?
3. What is the difference between host, source, and sourcetype?
4. What does the pipe character do in SPL?
5. How do AND, OR, NOT, and parentheses change a search?
6. Why is where commonly used after stats?
7. How does a universal forwarder differ from Splunk Enterprise?
8. What must be configured on the receiver before forwarded data can arrive?
9. When is a scheduled alert preferable to a real-time alert?
10. How does a report differ from an alert, and why should shared reports be tested with the intended permissions?

For continued study, review the Splunk Online Course and complete the Splunk activity.