VMware ESXi and vSphere Cluster Management

How to Create a Search Alert in Splunk

Learn how to save a Splunk search as an alert, configure thresholds, trigger modes, throttling, permissions, and actions, then test it in Triggered Alerts.

What a Splunk alert does

An alert is a saved search with trigger rules and one or more actions. Splunk runs or evaluates the search, checks whether its trigger condition is satisfied, and then performs the configured actions.

A saved search is a reusable search definition. It can be run manually, scheduled, used as a report, or configured as an alert.

Ad hoc search: a search you run interactively to investigate data. It does not automatically respond when future results meet a condition.
Report: a saved search intended to present results on a schedule or in a dashboard. A report does not necessarily have a firing condition or response action.
Alert: a saved search that evaluates a condition and performs actions when that condition is met.

Common alert uses include detecting an excessive event volume, a sudden increase in application errors, security indicators, failed processes, or an operational measurement that exceeds an acceptable threshold.

Start with a completed and validated search

Build and test the search before creating the alert. An alert can only be as useful as the search behind it.

Open Splunk Web and run the intended search.
Inspect the results and confirm that the search returns the events, statistics, or rows you expect.
Select the time range that represents the monitoring objective.
Run the search again with that time range and note the result count or relevant result fields.
When the search behaves as expected, open Save As and choose the option to create an alert.

The time range is important because the search result count depends on the window being evaluated. A search over the last 15 minutes can produce a very different result count from the same search over the last 300 days. For a scheduled alert, also consider how the selected time range relates to the execution schedule; overlapping windows can repeatedly evaluate the same events.

Most searches that can be run in Splunk Web can be saved as alerts, although available alert types and actions can vary by Splunk version, app, and permissions.

Identify the alert and set ownership

Give the alert a title that tells an operator what it detects. Avoid vague names such as Test Alert. A useful description states the monitored condition, scope, expected response, and owner when those details matter.

Title: API errors above the operational limit
Description: Monitors API error events in the production application. Investigate when the result count exceeds the configured limit. Owner: Platform Operations.

Set the alert's permissions in the app context where you are creating it:

Private: the alert is available only to you or the users permitted to access it.
Shared: the alert is made available to other users within the relevant Splunk app, subject to their roles and app access.

Sharing and editing are separate concerns. A user may be able to view or run a shared alert without being allowed to edit it. Roles, capabilities, app permissions, and administrative policy control what you can share, change, or execute.

Choose how the alert executes

Splunk commonly provides scheduled and real-time alert types. Choose based on how quickly the condition must be detected and how much search capacity the monitoring requires.

Characteristic	Scheduled alert	Real-time alert
When evaluation occurs	At specified intervals over a defined time range	Against incoming events continuously or with minimal delay
Typical latency	Usually related to the schedule interval	Usually lower, depending on data arrival and system load
Resource impact	Runs searches at scheduled times; overlapping or expensive searches can consume significant resources	Maintains continuous or near-continuous evaluation and can require more sustained resources
Best-fit use cases	Periodic health checks, volume thresholds, reports that need action, and most operational monitoring	Urgent detections where delay is operationally unacceptable
When to avoid	When even the schedule interval is too slow for the response requirement	When scheduled evaluation is sufficient or continuous processing would add unnecessary load

Prefer a scheduled alert unless real-time detection is operationally necessary. A reasonable schedule and time range can reduce resource consumption while still meeting the response target.

Configure the trigger condition

The trigger condition is the rule that determines whether the alert fires. A common condition uses the result count, meaning the number of search results returned for the selected time range.

For example, a condition can be configured as:

Metric: Number of returned search results
Operator: Greater than
Threshold: 100

This condition fires when the search returns more than 100 results—that is, at least 101 results. The outcome depends on three connected parts:

Search logic: filters, transformations, and aggregations determine what counts as a result.
Time range: determines which events are evaluated.
Threshold: determines how many returned results are required before firing.

Other alert conditions can use a custom conditional expression or a threshold based on a field in the search results. For instance, an alert might fire when a calculated error rate exceeds a limit, rather than when the raw number of rows exceeds a limit. Make sure the search produces the field or value required by the selected condition.

Choose the trigger mode

The trigger mode controls how many alert events and actions are produced when the condition is satisfied.

Trigger once: creates one alert event when the overall condition is met. This is usually appropriate when the condition represents one incident, such as an excessive number of errors.
Trigger for each result: creates separate alert activity for individual matching results. Use this when each result represents a distinct item that needs its own notification or downstream action.

Per-result triggering can produce a large number of notifications or integration requests. Use it only when individual-result handling is required, and ensure the search returns distinct, actionable results.

Use throttling to suppress repeats

A throttle is a suppression interval that limits repeated alert firings or actions after an alert fires. Throttling helps prevent a notification storm when the same condition remains true across multiple scheduled searches.

Choose a duration based on the event rate and the time operators need to respond. A short throttle may be suitable for rapidly changing conditions; a longer throttle may be better for a continuing incident where repeated notifications add no value. Test the duration because throttling can suppress repeated detections of the same condition.

Configuration	Effect on alert volume	Suitable scenario
Trigger once without throttling	One firing per evaluation in which the overall condition is met	Low-volume condition where every evaluation should be recorded
Trigger once with throttling	Repeated firings are suppressed during the throttle interval	Incident-style monitoring, such as sustained high error volume
Trigger for each result without throttling	Potentially one action for every matching result on every evaluation	Only carefully controlled, low-volume searches
Trigger for each result with throttling	Individual actions can still occur, but repeated activity is suppressed according to the throttle configuration	Distinct actionable records where duplicate actions must be limited

Configure triggered actions

A triggered action is the response performed when an alert fires. Configure one or more actions appropriate to the condition and the team responsible for responding.

Include the action that records the firing in the Triggered Alerts view. This provides a place to review alerts that have fired. Depending on your Splunk configuration and installed apps, other actions may include email or other notifications, webhooks, scripts, ticket creation, or integrations with external systems.

Some actions require administrator configuration, credentials, endpoints, scripts, or additional permissions. An alert can fire successfully even when a separate integration action fails, so verify the alert record and the delivery result independently.

Worked example: more than 100 results in 300 days

Suppose you need to monitor a search whose result count represents a long-term condition. A 300-day lookback is appropriate only when the monitoring goal requires it and the system can support the search cost.

Run and validate the search manually.
Set the time range to the required 300-day window.
Choose Save As, then choose the alert creation option.
Set a descriptive title and explain the monitored scope, expected response, and owner.
Choose scheduled execution unless the requirement truly calls for real-time detection.
Set the trigger condition to result count greater than 100.
Choose Trigger once so the condition creates one alert event rather than one event per result.
Set a throttle period appropriate to the response process.
Select the action that records the firing in Triggered Alerts. Add other approved actions if needed.
Save the alert.

The expected outcome is a fired alert entry when the search returns at least 101 results within the evaluated 300-day window. If the alert runs repeatedly with the same long window, throttling is especially important because the same historical results may continue to satisfy the condition.

Save, test, and verify

Save the alert only after reviewing its identity, app context, permissions, execution type, time range, condition, trigger mode, throttle, and actions.

To test it, use suitable test data or temporarily choose a safe, lower threshold that is known to be reachable. After the alert runs, review the Triggered Alerts area and confirm all of the following:

The alert evaluated the intended search and time range.
The condition was met at the expected result count or field value.
The alert appears in Triggered Alerts.
The selected notification or integration action executed.
Any failed action delivery has been investigated.

Restore the production threshold and throttle after testing. Avoid using a test configuration that can notify real recipients or create production tickets unintentionally.

Common alert problems

The alert never fires

Run the underlying search manually with the alert's exact time range. Compare its result count with the threshold. Also check whether the schedule has run, whether filtering or aggregation changes the expected count, and whether the alert has permission to access the required data. A temporary safe lower threshold can help confirm the configuration.

The alert fires too frequently

The threshold may be too low, the schedule may be too frequent, or per-result mode may be selected for a high-volume search. Review representative result volumes, increase the threshold or refine the search, use trigger once when individual actions are unnecessary, and set an appropriate throttle duration.

No entry appears in Triggered Alerts

Confirm that the trigger condition was met, that the alert was saved, and that the Triggered Alerts action was selected. Also check that you are viewing the correct app context and have permission to see the alert and its records.

Other users cannot see a shared alert

Review whether the alert is still private, whether it was shared in a different app context, and whether the other users have the required roles and app access. Sharing does not automatically grant every user editing capability.

Notifications or integrations do not run

First confirm that the alert fired. Then review the action configuration, required credentials, endpoints, scripts, permissions, throttle status, and relevant administrative logs. Test the action through an approved non-production method.

Configuration checklist

Setting	Purpose	Typical choices	Operational consideration
Title	Identifies the alert	Specific condition and scope	Make it understandable in a notification or incident list
Description	Explains what is monitored and what to do	Condition, scope, response, owner	Keep operational context with the alert
Permissions	Controls visibility and editing	Private or shared within the app	Roles and app context determine access
Alert type	Controls evaluation behavior	Scheduled or real-time	Prefer scheduled unless low latency is required
Time range or schedule	Defines the data window and execution timing	Recent interval, historical window, or defined schedule	Ensure the window matches the monitoring goal and capacity
Trigger condition	Determines when the alert fires	Result count, field threshold, or custom expression	Validate the search output before choosing the threshold
Trigger mode	Controls alert activity granularity	Once or for each result	Per-result mode can greatly increase action volume
Throttle	Suppresses repeated activity	Selected duration or no throttle	Match the interval to event rate and response time
Triggered actions	Defines the response	Triggered Alerts, notifications, webhooks, scripts, or tickets	Actions may require credentials, permissions, or administrator setup

For related guidance, see alert creation and management in this course.