VMware ESXi and vSphere Cluster Management

Create a Splunk Alert That Runs a Script

Learn how to create a real-time Splunk alert for DISK FULL events and run free_space.bat for each matching result.

A Splunk alert is a saved search with trigger conditions and one or more alert actions. In this example, a real-time search detects events containing DISK FULL, and a script action runs free_space.bat for each matching result.

Scripted alert actions can support controlled remediation, notification integrations, incident enrichment, or other operational responses. Because an alert can invoke a script repeatedly, treat the script as production automation: test it, restrict its permissions, and make its behavior safe when called more than once.

How the alert works

The end-to-end flow is:

1. An incoming event contains the disk-capacity error text.
2. A real-time search evaluates incoming events continuously.
3. The search produces a matching result.
4. A per-result trigger starts the configured alert action.
5. Splunk executes free_space.bat on the Splunk instance responsible for running the alert.

A real-time alert does not wait for the next scheduled search interval. This makes it useful for immediate event-driven handling, but continuous evaluation can consume more resources than a scheduled search. High event volume also increases the number of possible script invocations.

Validate the search condition

Begin with a search that identifies the events requiring action. The example criterion is the literal disk-full message:

"DISK FULL"

Run the search in the Search interface before creating the alert. Inspect the returned events and confirm that:

• The events come from the expected hosts or sources.
• The text represents a genuine disk-capacity failure.
• The search does not match unrelated messages.
• The result frequency is reasonable for a per-result action.

If your environment requires additional filtering, add the appropriate index, source, host, or field constraints and validate the narrower search. The alert should be based on a search that already produces the expected results; creating an alert does not correct an inaccurate search.

Save the search as an alert

1. Leave the completed search open in the Search interface.
2. Use the search save workflow and choose the option to create an alert, rather than saving a report.
3. Enter disk full alert as the alert title.
4. Enter a description such as Automated response for disk-capacity errors.
5. Configure the alert ownership and permissions.
6. Set the alert type and trigger behavior described in the next sections.
7. Configure the script action, review the settings, and save the alert.

A saved search is the stored search definition. It can be used as a report or as an alert. An alert adds trigger conditions and actions to that saved search, so selecting the alert workflow is important.

Configure ownership and permissions

Set the permission scope to Shared in App when other users or roles in the relevant Splunk app need to view, use, or manage the alert. This scope makes the knowledge object available within that app, subject to role permissions.

A private alert may be visible only to its owner. Sharing it in an app does not automatically grant every user full management rights. Users also need suitable role permissions for shared knowledge objects. Confirm both the app context and the roles that should administer the alert.

Configure a real-time, per-result alert

Choose Real-time

Set the alert type to Real-time. Splunk then continuously evaluates incoming events for matches instead of running the search only at scheduled intervals. This is appropriate when the response must begin as soon as a matching disk-full event arrives.

Monitor the resource impact of real-time searches. A broad search, a busy data source, or many simultaneous real-time alerts can increase search workload. Narrow the search to the relevant events and review event volume before enabling automated actions.

Choose Per-result

Set the trigger condition to Per-result. Each matching search result can trigger the alert action. In this example, each matching DISK FULL event can start the script.

Per-result triggering is intentionally immediate, but it can be noisy. If one failing host emits the same error repeatedly, the script may run many times. Inspect event frequency and consider aggregation, deduplication, throttling, suppression, or a different alert design when repeated execution is not appropriate.

Configure the script action

1. In the alert actions, select the option to run a script.
2. Enter free_space.bat as the script name.
3. Place the file in the Splunk script directory on the executing Splunk instance:

$SPLUNK_HOME/bin/scripts/free_space.bat

SPLUNK_HOME is the base installation directory of Splunk. The script directory is the location Splunk uses for alert scripts, typically $SPLUNK_HOME/bin/scripts.

The script must be available on the Splunk instance responsible for executing the alert. In a distributed deployment, placing the file on a search head or another host that does not execute the alert will not be sufficient. Confirm where the alert search and action run in your deployment.

Verify end-to-end behavior

1. Locate an existing event containing DISK FULL, or generate a controlled test event through an approved test source.
2. Confirm that the alert is enabled and saved with the real-time and per-result settings.
3. Wait for the matching event to be evaluated and confirm that the alert triggers.
4. Verify that free_space.bat starts on the correct Splunk instance.
5. Check the intended outcome, such as a controlled diagnostic or remediation result.
6. Review alert activity and the script's own logs or other logging evidence.

Use a test that cannot damage production data or services. The script should produce enough evidence to distinguish “the alert triggered” from “the script started” and from “the intended operation completed.”

Operational safety and maintenance

• Use least-privilege permissions for the script file and the Splunk service account.
• Avoid destructive, irreversible, or unbounded remediation actions.
• Make the script idempotent where possible, so repeated execution does not create progressively worse results.
• Validate inputs and handle missing hosts, unavailable dependencies, and partial failures.
• Document the alert owner, expected response, dependencies, logging location, and rollback behavior.
• Review event volume and alert activity after deployment.
• Retest the search and script after Splunk, operating-system, or application changes.

Troubleshoot common problems

The alert never runs the script

• Run "DISK FULL" manually and inspect whether matching events exist.
• Review the saved alert and confirm that it is enabled and configured as real-time.
• Confirm that the per-result condition is being met.
• Verify that the run-a-script action is selected and references the intended filename.

The script cannot be found or does not start

• Verify the expected path under $SPLUNK_HOME/bin/scripts.
• Check the spelling and capitalization of free_space.bat.
• Confirm that the script format matches the operating system and that the Splunk service account can read and execute it.
• Check script-side logs and the relevant alert activity for execution errors.

The script runs too often

• Inspect the number and frequency of matching events.
• Remember that per-result mode can invoke the action for every result.
• Consider deduplication, throttling, suppression, aggregation, or a less frequent alert design.
• Ensure that repeated invocation is safe and that the script is idempotent where practical.

Other users cannot view or manage the alert

• Review whether the alert is private instead of Shared in App.
• Confirm that users are working in the correct app.
• Check role permissions for access to shared knowledge objects and alert management.

Final configuration checklist

• The search returns the intended DISK FULL events.
• The object is saved as an alert, not only as a report.
• The title is disk full alert and the description explains its purpose.
• The permission scope is Shared in App when app-wide access is required.
• The alert type is Real-time.
• The trigger condition is Per-result.
• The action runs free_space.bat.
• The file exists at $SPLUNK_HOME/bin/scripts/free_space.bat on the executing instance.
• Permissions use least privilege.
• A controlled test confirms the event, alert, script start, and intended result.

For the complete worked example, use the create an alert that runs a script lesson as the reference for this configuration.