Create a Splunk Alert That Executes a Script When Disk Full Events Occur

Learn how to build a real-time Splunk disk-full alert that runs a server-side script, including search design, trigger settings, deployment, testing, security, and troubleshooting.

A Splunk alert is a saved search with trigger logic and one or more actions. When the search condition is met, Splunk can notify an operator or start an approved automation script. This lesson creates a real-time alert for disk-capacity failure messages and configures it to run a Windows batch script.

The example looks for the literal event text DISK FULL. In a production environment, an equivalent normalized field—such as a field indicating that a filesystem is full—may be more reliable than matching free-form text.

Plan the disk-full automation

The automation flow has four parts:

A search identifies events from the intended infrastructure logs.
The search is saved as an alert with a clear trigger condition.
The alert invokes the Run a Script action.
The script records the event and performs a carefully controlled notification or remediation.

For example, an operating system may write a message when a filesystem cannot accept more data. A matching event can start a script that gathers free-space information, sends a notification, or performs an approved cleanup operation. The script should not automatically delete files merely because an event contains the phrase DISK FULL.

Build and validate the search

Start in the Search app and construct the search before creating the alert. A basic search pattern is:

index=<target_index> sourcetype=<target_sourcetype> "DISK FULL"

Replace the placeholders with values from your environment. Add filters for the relevant host, source, or other fields when possible. For example, the final search might be scoped to operating-system logs from a known group of hosts rather than searching every event.

Use the time range picker to select a period that contains representative data. Identify the applicable:

Index: the data store containing the infrastructure logs.
Sourcetype: the event format or parser used for those logs.
Host: the system that generated the event.
Source: the file, input, or other origin of the event.
Time range: a period that includes known disk-full messages without unnecessarily scanning unrelated data.

Run the search and inspect the returned events. Confirm that the matches are genuine disk-capacity failures, not documentation, test text, or unrelated messages containing the same words. Do not save an alert until the underlying search returns the events you expect.

Prefer structured fields when available

Text matching is useful when the source only provides a message. If the data has a normalized field, use it instead. A field-based search could resemble:

index=<target_index> sourcetype=<target_sourcetype> disk_status="full"

The actual field name and value depend on the parser and data model. Field-based matching generally avoids variations in capitalization, punctuation, and message wording. See Splunk fields for related search concepts.

Save the search as an alert

With the validated search displayed, open the search page's save workflow and choose the option to save it as an alert.
Set the alert name to disk full alert.
Add a description such as: Initiates a disk-space response when a disk-full event is detected.
Choose the owner and app context deliberately.
Set the sharing scope, such as Shared in App, if other authorized users need to view or manage the alert.

Shared in App makes the alert available within the current Splunk app, subject to users' roles and capabilities. Ownership determines which user context owns the saved object, while the app context determines where it is managed and discovered. A user may be able to run a search but lack the capability to create, edit, share, or execute alert actions. Confirm these permissions before testing.

Configure a real-time alert

Set the alert type to Real-time. A real-time alert evaluates incoming events continuously or near continuously instead of waiting for a scheduled search interval. This is appropriate when disk exhaustion requires immediate handling.

Real-time searches consume search and system resources for as long as they remain active. Keep the search narrow by specifying the correct index, sourcetype, hosts, sources, and fields. Avoid an unrestricted search across all indexes when a targeted search is sufficient.

A scheduled alert may be a better choice when immediate handling is not required, when the environment must limit continuous search load, or when events should be grouped before action. A scheduled alternative can run the search frequently and trigger when the number of newly found disk-full events is greater than zero.

Choose the trigger granularity

Set the trigger mode to Per-result when every matching event should be considered an individual action. In this example, each qualifying event can invoke the script.

Per-result triggering has an important operational consequence: if one host emits many disk-full messages, Splunk may invoke the script many times. Use throttling or another suppression strategy to reduce duplicate actions during a recurring incident. A throttle can be keyed by a value such as host, when the alert configuration supports that grouping.

Per-result versus other alert trigger approaches

Per-result: Fires for each returned matching event. Best when each event represents a distinct item requiring action. Risk: repeated events can launch repeated scripts.

Number of results: Fires when the result count meets a configured condition. Best when a group of events should produce one response. Risk: a high threshold can delay action, while a low threshold can still create noise.

Once per search: Fires at most once when the search meets its condition. Best for one notification or one batch response. Risk: individual event context may be lost.

Scheduled threshold alert: Runs on a schedule and fires when a count or threshold is reached. Best when grouping, periodic evaluation, or lower continuous-search load matters. Risk: response time depends on the schedule.

Example alert configuration

Alert title: disk full alert

Description: Initiates a disk-space response when a disk-full event is detected.

Permission scope: Shared in App, if appropriate for the intended operators and administrators.

Alert type: Real-time

Trigger mode: Per-result

Trigger threshold or condition: Each returned event matching the search

Script action: Run a Script

Script filename: free_space.bat

Script directory: $SPLUNK_HOME\bin\scripts\free_space.bat

Optional throttle interval: An interval and grouping key chosen to prevent duplicate actions during one incident

Configure the Run a Script action

In the alert actions, select Run a Script. For the Windows example, choose or enter free_space.bat. Place the file at:

$SPLUNK_HOME\bin\scripts\free_space.bat

SPLUNK_HOME is the installation root of the Splunk instance. The bin\scripts directory is the designated location for scripts invoked by configured alert actions. The exact filesystem path depends on where Splunk was installed.

The script must be available locally to the Splunk instance that performs the alert action. Splunk passes alert context to the action according to the alert-action and Splunk version behavior. Treat all supplied arguments, environment values, and event-derived data as untrusted input: determine what context is actually available, validate it, and do not place raw event text directly into shell commands.

Deploy the script for the operating system

The example uses a Windows batch file because the deployment target is a Windows Splunk installation. Linux and Unix deployments typically use an executable shell script, Python program, or another supported executable format.

Filename and extension — Windows batch script: free_space.bat. Linux or Unix script: an executable file such as free_space.sh or a Python script.

Location under SPLUNK_HOME — Windows batch script: $SPLUNK_HOME\bin\scripts\free_space.bat. Linux or Unix script: $SPLUNK_HOME/bin/scripts/ followed by the script filename.

Execution account — Windows batch script: the Splunk service account, not necessarily an interactive administrator. Linux or Unix script: the account running the Splunk service.

Permissions — Windows batch script: the service account needs read and execute access and any narrowly scoped rights required by the action. Linux or Unix script: the file must be executable and readable by the service account, with ownership and permissions set according to local policy.

Interpreter or execution requirements — Windows batch script: use valid batch syntax and avoid assumptions about mapped drives or interactive profiles. Linux or Unix script: use a correct interpreter declaration where applicable, such as a shebang, and use absolute paths.

Logging destination — Windows batch script: write to a protected, known log location and capture errors. Linux or Unix script: use a protected log file or approved system logging facility.

In a distributed deployment, determine which component executes the alert action. Depending on the search and deployment design, this may be the search head, a designated alerting node, or another Splunk component. A script installed on an indexer is not automatically available to a search head. Deploy the file and its permissions to every relevant alert-executing instance, or use the platform's supported deployment mechanism.

Test and validate the complete workflow

Use a known disk-full event or a controlled test event that safely represents the condition.
Run the underlying search manually and confirm that the test event is returned.
Confirm that the alert is enabled, saved in the intended app, and owned and shared as expected.
Verify that the real-time trigger reaches the selected per-result condition.
Confirm that the script runs under the expected Splunk service account.
Check that the script produces the intended notification or safe remediation.
Review alert activity, Splunk scheduler or internal logs, and the script's own timestamped output, error output, and exit status.

Test the script independently with non-production-safe conditions before allowing it to change files or services. A useful execution record includes the time, host or event identifier, action taken, result, and error details without exposing secrets.

Security and reliability safeguards

Treat scripted alert actions as privileged automation. Review them like any other production change.
Use least privilege for the Splunk service account and for the operations performed by the script.
Validate and sanitize event-derived values before passing them to command interpreters, shell commands, or external tools.
Do not embed passwords, tokens, or other secrets directly in the script. Use approved secret management or protected configuration methods.
Make remediation idempotent: running it twice should not cause harmful repeated changes.
Use absolute paths and explicit timeouts instead of relying on an interactive user's environment.
Log successful and failed executions, return meaningful exit statuses, and handle errors explicitly.
Use throttling, deduplication, or host-based suppression when a source can emit repeated messages.
Document ownership, rollback steps, permissions, and change approval before enabling the action in production.

Troubleshoot common failures

The alert never fires

Run the underlying search over a period containing a known test event.
Check the index, sourcetype, host, source, field filters, and time behavior.
Confirm that the alert is enabled and visible in the intended app and owner context.
Verify that the real-time mode and trigger condition match the expected event behavior.
Review alert activity and scheduler-related internal logs.

The alert fires, but the script does not run

Verify the exact filename and path under SPLUNK_HOME\bin\scripts on the alert-executing instance.
Confirm that the selected script name matches the deployed file.
Test access and execution rights using the Splunk service account where permitted.
Check that the script was not installed only on an indexer when the search head performs the action.
Inspect Splunk internal logs, endpoint protection events, and script-generated logs for blocked execution.

The script runs too many times

Measure the frequency and count of matching events.
Remember that per-result mode processes every matching event.
Configure a throttle keyed by host or another identifying field, or switch to a count-based scheduled alert.
Make the script safe to execute repeatedly.

The script runs but remediation fails

Capture standard output, error output, exit codes, and timestamps.
Check whether the service account has the required, narrowly scoped privileges.
Replace assumptions about mapped drives, interactive profiles, and environment variables with explicit configuration and absolute paths.
Check remote-system availability and approved authentication methods.
Review the logic that identifies safe files, mounts, or disk locations before permitting cleanup.

Result

The completed configuration is a saved search named disk full alert, shared according to the required app permissions, running in real time, triggering per result, and using the Run a Script action to invoke free_space.bat. The action executes only where the script is deployed and accessible to the component and service account responsible for alert execution. In production, combine this configuration with throttling, least privilege, validation, logging, testing, and change control.