[CrowdStrike] Add Support of Crowdstrike Event Stream

packages/crowdstrike/_dev/build/docs/README.md

@@ -1,24 +1,28 @@
 # CrowdStrike Integration
 
-The [CrowdStrike](https://www.crowdstrike.com/) Falcon integration allows you to easily connect your CrowdStrike Falcon platform to Elastic for seamless onboarding of alerts and telemetry from CrowdStrike Falcon and Falcon Data Replicator. Elastic Security can leverage this data for security analytics including correlation, visualization and incident response. It provides support using three different modes for integrating CrowdStrike to the Elastic:
+The [CrowdStrike](https://www.crowdstrike.com/) integration allows you to easily connect your CrowdStrike Falcon platform to Elastic for seamless onboarding of alerts and telemetry from CrowdStrike Falcon and Falcon Data Replicator. Elastic Security can leverage this data for security analytics including correlation, visualization and incident response. It provides support using four different modes for integrating CrowdStrike to the Elastic:
 
-1. Falcon SIEM Connector: This is a pre-built integration designed to connect CrowdStrike Falcon with Security Information and Event Management (SIEM) systems. It streamlines the flow of security data from CrowdStrike Falcon to the SIEM, providing a standardized and structured way of feeding information into the SIEM platform. It includes the following datasets for receiving logs:
+1. **Falcon SIEM Connector**: This is a pre-built integration designed to connect CrowdStrike Falcon with Security Information and Event Management (SIEM) systems. It streamlines the flow of security data from CrowdStrike Falcon to the SIEM, providing a standardized and structured way of feeding information into the SIEM platform. It includes the following datasets for receiving logs:
 
 - `falcon` dataset: consists of endpoint data and Falcon platform audit data forwarded from [Falcon SIEM Connector](https://www.crowdstrike.com/blog/tech-center/integrate-with-your-siem/).
 
-2. CrowdStrike REST API: This provides a programmatic interface to interact with the CrowdStrike Falcon platform. It allows users to perform various operations such as querying information about unified alerts and hosts/devices. It includes the following datasets for receiving logs:
+2. **CrowdStrike REST API**: This provides a programmatic interface to interact with the CrowdStrike Falcon platform. It allows users to perform various operations such as querying information about unified alerts and hosts/devices. It includes the following datasets for receiving logs:
 
 - `alert` dataset: It is typically used to retrieve detailed information about unified alerts generated by the CrowdStrike Falcon platform, via Falcon Intelligence Alert API - `/alerts/entities/alerts/v2`.
 
 - `host` dataset: It retrieves all the hosts/devices in your environment providing information such as device metadata, configuration, and status generated by the CrowdStrike Falcon platform, via Falcon Intelligence Host/Device API - `/devices/entities/devices/v2`. It is more focused to provide the management and monitoring information of devices such as login details, status, policies, configuration etc.
 
-3. Falcon Data Replicator: This Collect events in near real time from your endpoints and cloud workloads, identities and data. CrowdStrike Falcon Data Replicator (FDR) enables you with actionable insights to improve SOC performance. FDR contains near real-time data collected by the Falcon platform's single, lightweight agent. It includes the following datasets for receiving logs:
+3. **Falcon Data Replicator**: This Collect events in near real time from your endpoints and cloud workloads, identities and data. CrowdStrike Falcon Data Replicator (FDR) enables you with actionable insights to improve SOC performance. FDR contains near real-time data collected by the Falcon platform's single, lightweight agent. It includes the following datasets for receiving logs:
 
 - `fdr` dataset: consists of logs forwarded using the [Falcon Data Replicator](https://github.com/CrowdStrike/FDR).
 
+4. **CrowdStrike Event Stream**: This streams security logs from CrowdStrike Event Stream, including authentication activity, cloud security posture management (CSPM), firewall logs, user activity, and XDR data. It captures real-time security events like user logins, cloud environment changes, network traffic, and advanced threat detections. The streaming integration provides continuous monitoring and analysis for proactive threat detection. It enhances visibility into user behavior, network security, and overall system health. This setup enables faster response capabilities to emerging security incidents. It includes the following datasets for receiving logs:
+
+- `falcon` dataset: consists of streaming data forwarded from CrowdStrike Event Stream.
+
 ## Compatibility
 
-This integration is compatible with both CrowdStrike Falcon SIEM-Connector-v2.0 and REST API.
+This integration is compatible with CrowdStrike Falcon SIEM-Connector-v2.0, REST API, and CrowdStrike Event Streaming.
 For Rest API support, this module has been tested against the **CrowdStrike API Version v1/v2**.
 
 ## Setup
@@ -35,6 +39,19 @@ For Rest API support, this module has been tested against the **CrowdStrike API
     | Alert         | read:alert    |
     | Host          | read:host     |
 
+### To collect data from CrowdStrike Event Stream, the following parameters from your CrowdStrike instance are required:
+
+1. Client ID
+2. Client Secret
+3. Token URL
+4. API Endpoint URL
+5. CrowdStrike App ID
+6. Required scopes for event stream:
+
+    | Data Stream   | Scope               |
+    | ------------- | ------------------- |
+    | Event Stream  | read: Event streams |
+
 ## Logs
 
 ### Alert

packages/crowdstrike/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.47.0"
+  changes:
+    - description: Add Support of CrowdStrike Event Stream.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11773
 - version: "1.46.0"
   changes:
     - description: Extract user and host names from the name field.

packages/crowdstrike/data_stream/falcon/_dev/deploy/docker/docker-compose.yml

@@ -6,3 +6,15 @@ services:
       - ./sample_logs:/sample_logs:ro
       - ${SERVICE_LOGS_DIR}:/var/log
     command: /bin/sh -c "cp /sample_logs/* /var/log/"
+  crowdstrike-streaming:
+    image: docker.elastic.co/observability/stream:v0.17.1
+    ports:
+      - 8080
+    volumes:
+      - ./files:/files:ro
+    environment:
+      PORT: 8080
+    command:
+      - http-server
+      - --addr=:8080
+      - --config=/files/config.yml

packages/crowdstrike/data_stream/falcon/_dev/deploy/docker/files/config.yml

@@ -0,0 +1,74 @@
+rules:
+  - path: /oauth2/token
+    methods: ["POST"]
+    query_params:
+      client_id: slightlysecretclientid
+      client_secret: totallysecretlientsecret
+      grant_type: client_credentials
+    request_headers:
+      Content-Type:
+        - "application/x-www-form-urlencoded"
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - "application/json"
+        body: |-
+          {{ minify_json `
+          {
+            "access_token": "topsecretaccesstokenthatshouldnotbeleakedforabit",
+            "token_type": "Bearer"
+          }
+          `}}
+
+  - path: /sensors/entities/datafeed/v2
+    methods: ["GET"]
+    request_headers:
+      authorization: ["Bearer topsecretaccesstokenthatshouldnotbeleakedforabit"]
+    query_params:
+      appId: nonsecretappid
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - "application/json"
+        body: |-
+          {{ minify_json `
+          {
+              "resources": [
+                  {
+                      "dataFeedURL": "http://svc-crowdstrike-streaming:8080/events",
+                      "sessionToken": {
+                          "token": "secretsessiontoken"
+                      },
+                      "refreshActiveSessionURL": "http://svc-crowdstrike-streaming:8080/refresh",
+                      "refreshActiveSessionInterval": 1800
+                  }
+              ]
+          }
+          `}}
+
+  - path: /events
+    methods: ["GET"]
+    request_headers:
+      authorization: ["Token secretsessiontoken"]
+    query_params:
+      offset: null
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - "application/json"
+        body: |-
+          {"metadata":{"customerIDString":"abcabcabc22221","offset":1,"eventType":"RemoteResponseSessionStartEvent","eventCreationTime":1698932494000,"version":"1.0"},"event":{"SessionId":"1111-fffff-4bb4-99c1-74c13cfc3e5a","HostnameField":"UKCHUDL00206","UserName":"[email protected]","StartTimestamp":1698932494,"AgentIdString":"fffffffff33333"}}
+          {"metadata":{"customerIDString":"abcabcabc22222","offset":2,"eventType":"RemoteResponseSessionStartEvent","eventCreationTime":1698932494000,"version":"1.0"},"event":{"SessionId":"1111-fffff-4bb4-99c1-74c13cfc3e5a","HostnameField":"UKCHUDL00206","UserName":"[email protected]","StartTimestamp":1698932494,"AgentIdString":"fffffffff33333"}}
+          {"metadata":{"customerIDString":"abcabcabc22223","offset":3,"eventType":"RemoteResponseSessionStartEvent","eventCreationTime":1698932494000,"version":"1.0"},"event":{"SessionId":"1111-fffff-4bb4-99c1-74c13cfc3e5a","HostnameField":"UKCHUDL00206","UserName":"[email protected]","StartTimestamp":1698932494,"AgentIdString":"fffffffff33333"}}
+
+  - path: /refresh
+    methods: ["POST"]
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - "application/json"
+        body: ''

packages/crowdstrike/data_stream/falcon/_dev/test/system/test-streaming-config.yml

@@ -0,0 +1,12 @@
+input: streaming
+service: crowdstrike-streaming
+data_stream:
+  vars:
+    url: http://{{Hostname}}:{{Port}}
+    client_id: slightlysecretclientid
+    client_secret: totallysecretlientsecret
+    token_url: http://{{Hostname}}:{{Port}}/oauth2/token
+    app_id: nonsecretappid
+    preserve_original_event: true
+assert:
+  hit_count: 3

packages/crowdstrike/data_stream/falcon/agent/stream/streaming.yml.hbs

@@ -0,0 +1,30 @@
+stream_type: crowdstrike
+url: {{url}}/sensors/entities/datafeed/v2
+auth:
+  client_id: {{client_id}}
+  client_secret: {{client_secret}}
+  token_url: {{token_url}}
+crowdstrike_app_id: {{app_id}}
+redact:
+  fields: ~
+program: |
+  state.response.decode_json().as(body,{
+    ?"cursor": body.?metadata.optMap(m, {"offset": m.offset}),
+    "events": [{
+          "message":  body.encode_json(),
+    }],
+  })
+tags:
+{{#if preserve_original_event}}
+  - preserve_original_event
+{{/if}}
+{{#each tags as |tag|}}
+  - {{tag}}
+{{/each}}
+{{#contains "forwarded" tags}}
+publisher_pipeline.disable_host: true
+{{/contains}}
+processors:
+{{#if processors}}
+{{processors}}
+{{/if}}

packages/crowdstrike/data_stream/falcon/manifest.yml

@@ -41,3 +41,70 @@ streams:
     template_path: log.yml.hbs
     title: Crowdstrike falcon logs (log)
     description: Collect Crowdstrike falcon logs using log input
+  - input: streaming
+    template_path: streaming.yml.hbs
+    title: CrowdStrike Falcon Logs
+    description: Collect Falcon logs from CrowdStrike Event Stream.
+    enabled: false
+    vars:
+      - name: url
+        type: text
+        title: URL
+        description: Base URL of the CrowdStrike API. Defaults to https://api.crowdstrike.com.
+        default: https://api.crowdstrike.com
+        required: true
+        show_user: true
+      - name: token_url
+        type: text
+        title: Token URL
+        description: Token URL of CrowdStrike.
+        default: https://api.crowdstrike.com/oauth2/token
+        required: true
+        show_user: false
+      - name: client_id
+        type: text
+        title: Client ID
+        description: Client ID for the CrowdStrike.
+        multi: false
+        required: true
+        show_user: true
+      - name: client_secret
+        type: password
+        title: Client Secret
+        description: Client Secret for the CrowdStrike.
+        multi: false
+        required: true
+        show_user: true
+        secret: true
+      - name: app_id
+        type: text
+        title: App ID
+        description: App ID for the CrowdStrike.
+        multi: false
+        required: true
+        show_user: true
+      - name: tags
+        type: text
+        title: Tags
+        multi: true
+        required: true
+        show_user: false
+        default:
+          - forwarded
+          - crowdstrike-falcon
+      - name: preserve_original_event
+        required: true
+        show_user: true
+        title: Preserve original event
+        description: Preserves a raw copy of the original event, added to the field `event.original`.
+        type: bool
+        multi: false
+        default: false
+      - name: processors
+        type: yaml
+        title: Processors
+        multi: false
+        required: false
+        show_user: false
+        description: >
+          Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.