Skip to main content

Deduplication Table

The Deduplication Table in the alert processing section is used to manage and configure deduplication rules for incoming alerts. Deduplication is the process of identifying and eliminating duplicate alerts to prevent redundant processing, allowing the analysts to focus on unique incidents.

This table allows administrators to customize deduplication rules for different alert sources, ensuring efficient alert management by consolidating repetitive alerts into single case. This reduces noise and helps maintain a clean and actionable alert feed within the case management system.

In the Deduplication Table, each entry specifies:

  • Alert Name: The name or identifier for the alert type or condition.
  • Product: The originating product or integration source of the alert.
  • Dedup Parameters: The parameters used to determine if alerts are duplicates (e.g., using observables like "URL" or "File Hash" to compare incoming alerts).
    • If you select the "None" attribute as the Dedup Parameters, it will disable the deduplication for the selected alert.
  • Rule Name: The specific rule applied for deduplication based on the parameters defined for each alert.
info

Please do not modify the table schema. Instead, simply add new records to the existing table schema.

Alert Name Syntax​

  • The alert name parameter is mutually exclusive with the other parameters in the Deduplication Table

  • If the Name parameter is selected as a deduplication parameter, it will cancel any other parameter and be used exclusively for the deduplication rule.

  • The % symbol can be used as a wildcard in alert names. This means that you can substitute % for any sequence of characters. This allows you to match multiple alert names without needing to specify each one exactly.

  • When the * symbol is used in alert names, it indicates that the deduplication rule must be applied to all alerts that include * in their name.

Thumbnail

How to Choose Dedup Parameters​

When selecting a deduplication parameter, consider the alert type and your preferred method of organizing and tracking alerts. The dedup parameter should group related alerts in a way that provides clear visibility and aligns with your desired approach. For Example:

  1. Malware Alerts: You can dedup using the File Hash and Hostname. This means each alert will represent a unique combination of malware and the affected host. It helps you clearly see which malware is impacting which device.

  2. Phishing Alerts: Dedup using the Sender Email Address. This combines all alerts from the same attacker, giving you a clear view of the broader phishing campaign instead of focusing on individual targets.

These examples illustrate common ways for choosing dedup parameters, but every user may have their own way of choosing dedup parameters. Some might prefer to dedup based on other criteria, such as IP address, domain, or other attributes specific to their own situation. This is why there is full flexibility in choosing the dedup parameter that fits best with your unique requirements.

info
  • You can select a single dedup parameter or multiple dedup parameters.
  • If you select the "None" attribute as the dedup parameters, it will disable the deduplication for the selected alert.

Key Considerations for Choosing Deduplication Parameters​

Based on our recommendations, here are the key factors to consider when selecting a dedup parameter:

  1. Relevance to the Alert Type

    • Choose parameters that are most relevant to the type of alert.
    • Example: For malware, attributes like File Hash or Hostname may be key, while for phishing, Sender Email Address might be more appropriate.

  2. Impact on Visibility

    • Consider how the dedup parameter will group alerts and how it affects the visibility of incidents.
    • Does it provide the level of detail you need, or does it oversimplify the data?

  3. Prioritize Useful Information

    • Choose parameters that highlight key actions or insights.
    • Example: Deduping by IP Address might show attack sources, whereas deduping by Target Email Address could focus on affected users.

  4. Scalability

    • Ensure the parameter works well for large volumes of alerts.
    • If the deduplication criteria are too narrow, you may still end up with too many grouped alerts.

  5. Flexibility for Custom Approaches

    • Remember that each organization may have unique requirements based on their environment, tools, and workflows. Choose a parameter that best fits your requirements.

  6. Finding the Right Grouping Criteria

    • Avoid making the grouping too broad or too narrow.
    • Example: Deduping only by Hostname might merge unrelated alerts, while using too many parameters might fragment related alerts.

  7. Context of the Security Environment

    • Consider factors like industry, threat landscape, and compliance needs.
    • Example: In a regulated industry, dedup parameters may need to highlight compliance breaches clearly.

  8. Simplify Analysis

    • Ensure the dedup parameter simplifies analysis rather than complicating it.
    • Grouping should make patterns and trends easier to spot.

  9. Alignment with Incident Prioritization

    • Choose parameters that align with how incidents are prioritized.
    • Example: Grouping alerts by Severity or Attack Vector might help focus on critical threats first.

  10. Historical Trends

    • Consider what dedup parameters have worked well in the past for similar alert types.

How to Create a Deduplication Rule​

  1. In the left-hand sidebar, click on the "Tables" and select the "Deduplication Rules" tab at the top.
Thumbnail
  1. Click on the blue "New Record" button located in the top-right corner of the page.
Thumbnail
  1. A pop-up will appear where you can configure the new rule. Complete the following fields:
  • Alert Name: Enter the name of the alert to which this rule will apply.
  • Product: Specify the product or service the rule targets- e.g., Microsoft Outlook, Crowdstrike or Proofpoint .
  • Dedup Parameters: Select the deduplication parameters.
    • You can select a single dedup parameter or multiple dedup parameters.
    • If you select the "None" attribute as the Dedup Parameter, it will disable the deduplication for the selected alert
  • Rule Name: Provide a unique name for the rule.
Thumbnail

  1. After filling in the required fields, click the Save button at the bottom of the form. Your new rule will appear in the table, listed alongside existing deduplication rules.

  2. The Get Duplication Rule Subflow will automatically select the most suitable deduplication rule for each incoming alert. No changes need to be made to the Get Duplication Rule Subflow.

  • It determines the appropriate rule by following this priority:

    • First priority: A deduplication rule that matches the alert name is applied.
    • Second priority: If no alert-name match is found, the subflow looks for a deduplication rule associated with the alert's product name.
    • Fallback rule: If neither a name nor product-based rule exists, a default deduplication rule is created and applied automatically based on the alert name.

  • This ensures that alerts are always processed with the most relevant deduplication rule, minimizing manual intervention.

Thumbnail