Deduplication Rules
Deduplication Rules define how the system detects and groups repeated alerts that represent the same underlying issue. Instead of creating a new case for every incoming alert, the system uses these rules to decide whether an alert should be attached to an existing case.
This reduces alert fatigue, prevents redundant case creation, and helps analysts focus on investigating unique incidents. Users can configure deduplication rules based on fields like alert type, source, severity, or custom attributes — allowing for precise control over how alerts are consolidated into cases.
Tip: Consider the Following Questions
Tip: Consider the Following Questions
When reviewing the vendor and its sample alert payload, consider the following questions:
- Are the field names and structures consistent across different alert types from this vendor?
- Does the context or content of the alerts vary significantly between types?
- Can you reasonably estimate how many alerts will be generated by the rules you’re considering?
Your answers will help determine the best rule configuration strategy. In some cases, it may be appropriate to create specific rules tailored to each alert type. In others, a single generic rule may suffice. You can also combine both approaches—define specific rules for well-understood alert types and use a generic rule as a fallback.
In this setup, the system will prioritize specific rules. The generic rule will only be applied when no specific rule matches the alert, effectively serving as a catch-all fallback.
How to Create a Deduplication Rule
Navigate to Case Management Settings
In Case Management Settings click on the “Deduplication Rules” tab, under the General Settings section.
Click 'New Rule' button
located in the top-right corner of the page.
Fill out the following parameters:
| Parameter | Description |
|---|---|
| Rule Name | Provide a unique name for the deduplication rule. |
| Description | Provide a brief description for the deduplication rule |
| Vendors | Specify the vendor the rule targets- e.g., Microsoft Outlook, Crowdstrike or Proofpoint . You can select one or multiple vendors. If no vendors are selected, all vendors from the list will be included by default. |
| Alert Name | Enter the name of the alert to which this rule will apply. |
| Use Regex | Use regex to indicate a matching criteria. |
| Lookback Period | The time frame, in days, during which past cases will be considered as duplicates for the specified rule. |
| Deduplication Condition | The condition for applying the deduplication rule |
Save the Newly Created Rule
After filling in the required fields, click the Save button. Your new deduplication rule will appear in the table, listed alongside existing deduplication rules.
Deduplication Rule Assignment
The Case Deduplication Action will then automatically select the most suitable deduplication rule for each incoming alert.
Alert Name Syntax Rules
-
If the ‘Alert Name’ input is left blank, the ‘Extract Observable Rules’ will be applied to all alerts.
-
When matching alert names, the system prioritizes the longest matching string. Exact matches take precedence over regular expression (regex) patterns.
-
The Alert Name field in ‘Extract Observable Rule’ supports Regular Expressions (Regex), allowing you to define flexible matching patterns. This enables dynamic grouping and processing of similar alerts — even when alert names contain varying formats, dynamic elements like timestamps or IDs, or share common prefixes or suffixes.
Example of Using Regex in an Alert Name
Example of Using Regex in an Alert Name
If we want to match all alert names that start with “Critical Alert” followed by a series of numbers.
Regex Use Case Example
Explanation:
^matches the beginning of the alert name.Critical Alertmatches the literal text.[0-9]+matches one or more digits.$matches the end of the alert name. This would match:Critical Alert 123Critical Alert 4567
But not:
Critical Alert(no numbers)Info Alert 123(doesn’t start with “Critical Alert”)
Deduplication Conditions
Deduplication Conditions let you define the logic used to detect duplicate alerts, observables or relations.
Each deduplication rule can include multiple conditions, and these conditions are combined using the AND operator — meaning all conditions in the rule must be true for a match.
When you define multiple rules, they are combined using the OR operator — meaning if any one rule matches, the alert is considered a duplicate.
- To mark something as a duplicate only when all specific criteria are met, include all those conditions in a single rule.
- To mark something as a duplicate if it meets any one of several criteria, create a separate rule for each.
Condition Statement Options
-
When a case contains an
- Alert with the same name: This condition checks if an incoming alert has the same name as an alert already linked to an existing case.
-
When a case contains:
- all observables of all types and any relation: This condition checks if all observables (regardless of type or relation) from the incoming alert are already present in a case. If even one observable is missing, the condition won’t match.
- all observables of type (X) and relation (Y): This condition checks whether all observables of a certain type (e.g., IP Address, Hash, Domain) and with a specific relation (e.g., Source, Destination, Executed By) from the alert are already linked to an existing case. If all observables that meet both the type and relation match are found in the case, it is considered a duplicate.
-
When a case contains:
- some observables any type and any relation -This condition checks whether at least one observable — regardless of its type or relation — from the alert already exists in a case.
- some observables of type (X) and relation (Y): This condition checks checks if at least one observable of a specific type and relation from the incoming alert is already in an existing case.
Then: (If all above conditions are satisfied)link the alert and its observables to that case.
Deduplication Rule Use Case Examples
Use Case Example: Deduplication Rule for Okta Brute Force Alerts
Use Case Example: Deduplication Rule for Okta Brute Force Alerts
Rule Configuration
- Rule Name: Default Okta
- Vendor: Okta
- Alert Name: Brute Force Attack Detected
- Use Regex: Enabled
- Lookback Period: 15 days
Deduplication Condition
Scenario
At 14:00, the system receives an alert from Okta with the name “Brute Force Attack Detected”. The alert includes a source IP address of 192.168.1.10. Since there is no existing case that matches, a new case is created.
At 14:10, another alert with the same name and same source IP is received. Because the deduplication rule is configured to:
- Match alerts by name (with regex support),
- Compare observables of type IP Address with a Source relation,
the system identifies that the existing case from 14:00 meets all criteria. As a result, the new alert is automatically linked to the existing case instead of creating a new one.
Outcome
This deduplication rule prevents case sprawl caused by repeated alerts for the same issue. It helps streamline investigation by consolidating related events into a single case, reducing alert fatigue and improving analyst efficiency.
Use Case Example: Deduplication Rule for Duplicate File Alerts in ServiceNow
Use Case Example: Deduplication Rule for Duplicate File Alerts in ServiceNow
Rule Configuration
- Rule Name: Duplicate File Alert
- Vendor: ServiceNow
- Alert Name: Duplicate File Alert
- Use Regex: Disabled
- Lookback Period: 15 days
Deduplication Condition
Scenario
On Monday at 10:00 AM, a “Duplicate File Alert” is received from ServiceNow, containing a file hash observable (e.g., 098f6bcd4621d373cade4e832627b4f6) with a relation of IP Destination. Since no case exists with this specific hash, a new case is created.
Two days later at 2:00 PM, a second “Duplicate File Alert” is received from ServiceNow, again referencing the same file hash with the IP Destination relation. Because the deduplication rule is set to:
- Match the alert name using regex,
- Look for any hash observables with the relation IP Destination,
The system identifies that the previous case contains the same hash observable. As a result, the new alert is automatically linked to the existing case instead of creating a new one.
Outcome
This deduplication rule ensures that alerts referencing the same suspicious file are grouped together, even if they are triggered from different IP destinations or at different times. It reduces case duplication, improves alert triage efficiency, and helps analysts maintain context over the lifespan of a threat.
Key Considerations for Choosing Deduplication Parameters
Key Considerations for Choosing Deduplication Parameters
Based on our recommendations, here are the key factors to consider when creating deduplication conditions:
-
Relevance to the Alert Type
- Choose conditions that are most relevant to the type of alert.
- Example: For malware, attributes like File Hash or Hostname may be key, while for phishing, Sender Email Address might be more appropriate.
-
Impact on Visibility
- Consider how the deduplication condition will group alerts and how it affects the visibility of incidents.
- Does it provide the level of detail you need, or does it oversimplify the data?
-
Prioritize Useful Information
- Choose parameters that highlight key actions or insights.
- Example: Deduping by IP Address might show attack sources, whereas deduping by Target Email Address could focus on affected users.
-
Scalability
- Ensure the parameter works well for large volumes of alerts.
- If the deduplication criteria are too narrow, you may still end up with too many grouped alerts.
-
Flexibility for Custom Approaches
- Remember that each organization may have unique requirements based on their environment, tools, and workflows. Choose a parameter that best fits your requirements.
-
Finding the Right Grouping Criteria
- Avoid making the grouping too broad or too narrow.
- Example: Deduping only by Hostname might merge unrelated alerts, while using too many parameters might fragment related alerts.
-
Context of the Security Environment
- Consider factors like industry, threat landscape, and compliance needs.
- Example: In a regulated industry, deduplication conditions may need to highlight compliance breaches clearly.
-
Simplify Analysis
- Ensure the deduplication condition simplifies analysis rather than complicating it.
- Grouping should make patterns and trends easier to spot.
-
Alignment with Incident Prioritization
- Choose parameters that align with how incidents are prioritized.
- Example: Grouping alerts by Severity or Attack Vector might help focus on critical threats first.
-
Historical Trends
- Consider what dedup parameters have worked well in the past for similar alert types.