[crowdstrike] Add support to parse the custom notification format pro…

…vided by Crowdstrike (elastic#2198)

packages/crowdstrike/_dev/build/docs/README.md

@@ -22,8 +22,38 @@ Contains endpoint data and CrowdStrike Falcon platform audit data forwarded from
 
 ### FDR
 
-The Falcon Data Replicator replicates log data from your CrowdStrike environment to a stand-alone target. This target can be a location on the file system, or an S3 bucket.
+The CrowdStrike Falcon Data Replicator (FDR) allows CrowdStrike users to replicate FDR data from CrowdStrike 
+managed S3 buckets. CrowdStrike writes notification events to a CrowdStrike managed SQS queue when new data is 
+available in S3.
 
+This integration can be used in two ways. It can consume SQS notifications directly from the CrowdStrike managed 
+SQS queue or it can be used in conjunction with the FDR tool that replicates the data to a self-managed S3 bucket 
+and the integration can read from there.
+
+In both cases SQS messages are deleted after they are processed. This allows you to operate more than one Elastic 
+Agent with this integration if needed and not have duplicate events, but it means you cannot ingest the data a second time.
+
+#### Use with CrowdStrike managed S3/SQS
+
+This is the simplest way to setup the integration, and also the default.
+
+You need to set the integration up with the SQS queue URL provided by Crowdstrike FDR.
+Ensure the `Is FDR queue` option is enabled.
+
+#### Use with FDR tool and data replicated to a self-managed S3 bucket
+
+This option can be used if you want to archive the raw CrowdStrike data.
+
+You need to follow the steps below:
+
+- Create a S3 bucket to receive the logs.
+- Create a SQS queue.
+- Configure your S3 bucket to send object created notifications to your SQS queue.
+- Follow the [FDR tool](https://github.com/CrowdStrike/FDR) instructions to replicate data to your own S3 bucket.
+- Configure the integration to read from your self-managed SQS topic.
+- Disable the `Is FDR queue` option in the integration.
+
+**NOTE: While the FDR tool can replicate the files from S3 to your local file system, this integration cannot read those files because they are gzip compressed, and the log file input does not support reading compressed files.**
 
 #### Configuration for the S3 input
 

packages/crowdstrike/changelog.yml

@@ -1,4 +1,12 @@
 # newer versions go on top
+- version: "1.0.4"
+  changes:
+    - description: Add ability to read from both FDR provided and user owned SQS queues for FDR.
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/2198
+    - description: Pipeline fixes for FDR
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/2198
 - version: "1.0.3"
   changes:
     - description: Uniform with guidelines

packages/crowdstrike/data_stream/fdr/_dev/test/pipeline/test-fdr.log

@@ -122,3 +122,4 @@
 {"AuthenticationId":"703298","ConfigBuild":"1007.3.0011603.1","ConfigStateHash":"2642284486","ContextProcessId":"1161025471861","ContextThreadId":"34929528116709","ContextTimeStamp":"1604851030.593","DiskParentDeviceInstanceId":"USB\\VID_1058\u0026PID_2621\\57583431453939315A4C5255","EffectiveTransmissionClass":"3","Entitlements":"15","FileEcpBitmask":"0","FileIdentifier":"262fbc677256cf4c8d6c6a227285a072c06830873b000000","FileObject":"18446664963104449168","IrpFlags":"1028","IsOnNetwork":"0","IsOnRemovableDisk":"1","MajorFunction":"18","MinorFunction":"0","OperationFlags":"0","Size":"517029","TargetFileName":"\\Device\\HarddiskVolume5\\01.png.tmp$$","TokenType":"1","UserName":"user9","aid":"ffffffff16bf4c7bb5ad755a4722025c","aip":"208.216.134.196","cid":"ffffffff30a3407dae27d0503611022d","event_platform":"Win","event_simpleName":"GenericFileWritten","id":"ffffffff-1111-11eb-800a-06cecfd73923","name":"GenericFileWrittenV11","timestamp":"1604851031298"}
 {"ConfigBuild":"1007.3.0011603.1","ConfigStateHash":"666346415","ContextProcessId":"1717987648455","ContextThreadId":"55064470042288","ContextTimeStamp":"1604850899.164","EffectiveTransmissionClass":"3","Entitlements":"15","VolumeName":"\\Device\\HarddiskVolume27","aid":"ffffffff896b43725b83c79aa79959da","aip":"208.216.150.196","cid":"ffffffff30a3407dae27d0503611022d","event_platform":"Win","event_simpleName":"FsVolumeUnmounted","id":"ffffffff-1111-11eb-9f70-0634389d9ea9","name":"FsVolumeUnmountedV2","timestamp":"1604850899812"}
 {"ConfigBuild":"1007.4.0009906.1","ConfigStateHash":"3429017943","ContextProcessId":"66426035996442255","ContextTimeStamp":"1604851098.548","Entitlements":"15","aid":"ffffffff899541b94b9adff8922aa70a","aip":"208.193.200.164","cid":"ffffffff30a3407dae27d0503611022d","event_platform":"Mac","event_simpleName":"FirewallDisabled","id":"ffffffff-1111-11eb-9d4c-02f402df8c1f","name":"FirewallDisabledMacV1","timestamp":"1604851040625"}
+{"AgentLoadFlags":"0","AgentLocalTime":"1636436839.9529998","AgentTimeOffset":"125.319","AgentVersion":"6.31.14404.0","BiosManufacturer":"Apple Inc.","BiosVersion":"1554.140.20.0.0 (iBridge: 18.16.14759.0.1,0)","ChassisType":"Laptop","City":"San Francisco","ComputerName":"mac1","ConfigBuild":"1007.4.0014404.1","ConfigIDBuild":"14404","Continent":"North America","Country":"United States","FalconGroupingTags":"-","FirstSeen":"1625682391.0","HostHiddenStatus":"Visible","MachineDomain":"none","OU":"none","PointerSize":"none","ProductType":"1","SensorGroupingTags":"-","ServicePackMajor":"none","SiteName":"none","SystemManufacturer":"Apple Inc.","SystemProductName":"MacBookPro16,2","Time":"1636448427.3539999","Timezone":"America/Los_Angeles","Version":"Big Sur (11.0)","aid":"fffffffffffaaaaaaaaabbbbbbbb","aip":"208.30.227.225","cid":"ffffffff30a3407dae27d0503611022ff","event_platform":"Mac"}