Subflow 1- Extract Observables
The Subflow 1 - Extract Observables is a foundational component in the On New Table Record - Process Alert automated workflow. Its primary role is to parse incoming alert data from the payload and extract key observables, which are essential data points like IP addresses
, URLs
, usernames
, file hashes
, and hostnames
. These observables are critical for understanding the context and impact of an alert, enabling subsequent enrichment, triage, and response actions.
How It Works
Initialization and Preprocessing:
- The subflow begins by initializing an observable list to store extracted data points. This ensures all relevant observables are collected systematically and can be processed further.
Template-Based Parsing:
Using templates from the Alert Template Table, the subflow identifies the specific paths in the alert payload where observables are located. These templates are defined in JSON format, with each key representing a payload path (e.g.,
device.external_ip
) and its corresponding value indicating the observable type (e.g., "IP Address").For example, if an alert contains a field
device.hostname
, the template maps it as"Target Hostname"
, ensuring consistent extraction across alerts.
Extraction and Validation:
- The system converts data into a standardized format (e.g., lowercase for uniformity) and uses the JSON templates to extract observables. It then checks whether each extracted observable is valid and unique, ensuring no duplicate or irrelevant data is processed.
Appending Observables:
- Extracted observables are appended to the initialized list, categorized by type (e.g., network observables like
IPs
, user observables likeusernames
)`. The system dynamically updates the list with valid entries, which is critical for downstream tasks.
- Extracted observables are appended to the initialized list, categorized by type (e.g., network observables like
Post-Processing:
- After extraction, any unstructured or unused identifiers are removed from the observable list to maintain efficiency and relevance. This ensures that only actionable data is passed on to the next stages.
Example of Raw Alert Payload
Unfortunately, Markdown doesn't natively support collapsible elements without additional CSS or JavaScript. However, some Markdown renderers (like GitHub, GitLab, or some note-taking apps) allow you to create collapsible sections using a specific syntax.
For instance, you can use the following approach, which uses the "details" and "summary" HTML tags to achieve collapsibility. This works in some Markdown environments that support HTML rendering (such as GitHub).
Click to Expand the Raw Alert Payload Example
{
"agent_id": "88a37ca1562e4abc800f4e548e83f899",
"aggregate_id": "aggind:bc44a7a3e758465f857867dcf4ac8c17:565469335392665337",
"alleged_filetype": "rtf",
"cid": "5686234480df4be090fbcf044a2708d4",
"cloud_indicator": false,
"cmdline": "./whoami.rtf",
"composite_id": "5686234480df4be090fbcf044a2708d4:ind:bc44a7a3e758465f857867dcf4ac8c17:565469335361356727-145-335777",
"confidence": 50,
"context_timestamp": "2024-07-24T15:25:00.113Z",
"control_graph_id": "ctg:bc44a7a3e758465f857867dcf4ac8c17:565469335392665337",
"crawled_timestamp": "2024-07-24T19:24:30.660636531Z",
"created_timestamp": "2024-07-24T15:26:04.807574447Z",
"data_domains": ["Endpoint"],
"description": "An executable was run with a contradicting file extension",
"device": {
"agent_load_flags": "0",
"agent_local_time": "2024-07-16T13:16:41.364Z",
"agent_version": "7.17.18604.0",
"cid": "5686234480df4be090fbcf044a2708d4",
"config_id_base": "65994763",
"config_id_build": "18604",
"config_id_platform": "4",
"device_id": "bc44a7a3e758465f857867dcf4ac8c17",
"external_ip": "180.112.146.250",
"first_seen": "2024-05-16T12:50:44Z",
"hostname": "DESKTOP-HLLEERH",
"last_seen": "2024-07-24T14:58:25Z",
"local_ip": "10.200.200.4",
"mac_address": "b0-de-28-07-15-64",
"major_version": "21",
"minor_version": "3",
"modified_timestamp": "2024-07-24T15:22:09Z",
"os_version": "Monterey (12)",
"platform_id": "1",
"platform_name": "Mac",
"product_type_desc": "Workstation",
"status": "normal",
"system_manufacturer": "Apple Inc.",
"system_product_name": "MacBookPro18,1"
},
"display_name": "FalseExecutableExtension",
"documents_accessed": [
{
"filename": "dtracehelper",
"filepath": "/dev/",
"timestamp": "1721834700"
}
],
"email_sent": true,
"falcon_host_link": "https://falcon.us-2.crowdstrike.com/activity-v2/detections/5686234480df4be090fbcf044a2708d4:ind:bc44a7a3e758465f857867dcf4ac8c17:565469335361356727-145-335777?_cid=g04000ul2m72oytvdbnm7n4e6bu5pkj4",
"filename": "whoami.rtf",
"filepath": "/Users/wilder/whoami.rtf",
"files_accessed": [
{
"filename": "dtracehelper",
"filepath": "/dev/",
"timestamp": "1721834700"
}
],
"global_prevalence": "common",
"incident": {
"created": "2024-07-24T15:24:36Z",
"end": "2024-07-24T15:26:01Z",
"id": "inc:bc44a7a3e758465f857867dcf4ac8c17:e6cb706ea495450fb779ad0a1e084bda",
"score": "19.15170747011056",
"start": "2024-07-24T15:24:36Z"
},
"indicator_id": "ind:bc44a7a3e758465f857867dcf4ac8c17:565469335361356727-145-335777",
"ioc_values": [],
"local_prevalence": "common",
"local_process_id": "44019",
"md5": "32ff28d4fdb4b244c355d7f8378fa2b1",
"name": "FalseExecutableExtension",
"objective": "Keep Access",
"parent_details": {
"cmdline": "login -pf wilder",
"filename": "login",
"filepath": "/usr/bin/login",
"local_process_id": "682",
"md5": "0e4f66991f4bfd0e96e5d28b52460ebf",
"process_graph_id": "pid:bc44a7a3e758465f857867dcf4ac8c17:565244152166793222",
"process_id": "565244152166793222",
"sha256": "178ba564b39bd07577e974a9b677dfd86ffa1f1d0299dfd958eb883c5ef6c3e1",
"timestamp": "1601-01-01T00:00:00.000Z",
"user_name": "root"
},
"pattern_disposition": 0,
"pattern_disposition_description": "Detection, standard detection.",
"platform": "Mac",
"severity": 50,
"severity_name": "Medium",
"timestamp": "2024-07-24T15:25:00.395Z",
"type": "ldt"
}
Examples of Extracted Observables
{
"agent_id": "Device Agent ID",
"device.external_ip": "IP Address",
"device.local_ip": "IP Address",
"device.hostname": "Hostname",
"user_name": "Username",
"sha256": "File Hash",
"user_principal": "Username",
"parent_details.sha256": "File Hash"
}
- Network Observables:
device.external_ip
: Identifies an external IP address potentially involved in an alert.device.local_ip
: Captures local IPs within the organization's network for context.
- Host and User Observables:
device.hostname
: Indicates the specific host impacted by the alert.user_name
: Extracts usernames related to suspicious activity.
- File-Based Observables:
sha256
: Extracts file hashes to identify potentially malicious files.parent_details.sha256
: Retrieves parent process file hashes for advanced analysis.
Significance in the Workflow
This subflow ensures that all actionable data is prepared for subsequent stages in the workflow. For instance:
- The Enrich Observables subflow leverages these extracted data points to gather additional threat intelligence.
- The Check for Case Duplicates subflow uses these observables to identify if similar alerts already exist in the system, preventing redundancy.
By structuring the raw alert data into meaningful and categorized observables, Subflow 1 - Extract Observables acts as a cornerstone in the efficient and accurate processing of security alerts, laying the groundwork for automated responses and decision-making.