[Microsoft_Defender_Cloud] Initial Release for the Microsoft Defender…

… for Cloud (#6593)

* Initial Release

* Update the changelog entry

* Indentified Readme changes from comments on other integration and resolved

* Resolve review comments

* Conflicts Resolve

* Review comments resolve

.github/CODEOWNERS

@@ -142,6 +142,7 @@
 /packages/mattermost @elastic/security-external-integrations
 /packages/memcached @elastic/obs-infraobs-integrations
 /packages/microsoft @elastic/security-external-integrations
+/packages/microsoft_defender_cloud @elastic/security-external-integrations
 /packages/microsoft_defender_endpoint @elastic/security-external-integrations
 /packages/microsoft_dhcp @elastic/security-external-integrations
 /packages/microsoft_exchange_online_message_trace @elastic/security-external-integrations

packages/microsoft_defender_cloud/_dev/build/build.yml

@@ -0,0 +1,4 @@
+dependencies:
+  ecs:
+    reference: [email protected]
+    import_mappings: true

packages/microsoft_defender_cloud/_dev/build/docs/README.md

@@ -0,0 +1,72 @@
+# Microsoft Defender for Cloud
+
+The [Microsoft Defender for Cloud](https://learn.microsoft.com/en-us/azure/defender-for-cloud/defender-for-cloud-introduction) integration allows you to monitor security alert events. When integrated with Elastic Security, this valuable data can be leveraged within Elastic for analyzing the resources and services that users are protecting through Microsoft Defender.
+
+Use the Microsoft Defender for Cloud integration to collect and parse data from **Azure Event Hub** and then visualize that data in Kibana.
+
+## Data streams
+
+The Microsoft Defender for Cloud integration collects one type of data: event.
+
+**Event** allows users to preserve a record of security events that occurred on the subscription, which includes real-time events that affect the security of the user's environment. For further information connected to security alerts and type, Refer to the page [here](https://learn.microsoft.com/en-us/azure/defender-for-cloud/alerts-reference).
+
+## Prerequisites
+
+To get started with Defender for Cloud, user must have a subscription to Microsoft Azure.
+
+## Requirements
+
+- Elastic Agent must be installed.
+- You can install only one Elastic Agent per host.
+- Elastic Agent is required to stream data from the **Azure Event Hub** and ship the data to Elastic, where the events will then be processed via the integration's ingest pipelines.
+
+### Installing and managing an Elastic Agent:
+
+You have a few options for installing and managing an Elastic Agent:
+
+### Install a Fleet-managed Elastic Agent (recommended):
+
+With this approach, you install Elastic Agent and use Fleet in Kibana to define, configure, and manage your agents in a central location. We recommend using Fleet management because it makes the management and upgrade of your agents considerably easier.
+
+### Install Elastic Agent in standalone mode (advanced users):
+
+With this approach, you install Elastic Agent and manually configure the agent locally on the system where it’s installed. You are responsible for managing and upgrading the agents. This approach is reserved for advanced users only.
+
+### Install Elastic Agent in a containerized environment:
+
+You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry, and we provide deployment manifests for running on Kubernetes.
+
+There are some minimum requirements for running Elastic Agent and for more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
+
+The minimum **kibana.version** required is **8.3.0**.
+
+## Setup
+
+### To collect data from Microsoft Azure Event Hub, follow the below steps:
+
+- Configure the Microsoft Defender for Cloud on Azure subscription. For more detail, refer to the link [here](https://learn.microsoft.com/en-us/azure/defender-for-cloud/get-started).
+
+### Enabling the integration in Elastic:
+
+1. In Kibana, go to Management > Integrations.
+2. In the "Search for integrations" search bar, type Microsoft Defender for Cloud.
+3. Click on the "Microsoft Defender for Cloud" integration from the search results.
+4. Click on the Add Microsoft Defender for Cloud Integration button to add the integration.
+5. While adding the integration, if you want to collect logs via **Azure Event Hub**, then you have to put the following details:
+   - eventhub
+   - consumer_group
+   - connection_string
+   - storage_account
+   - storage_account_key
+   - storage_account_container (optional)
+   - resource_manager_endpoint (optional)
+
+## Logs reference
+
+### Event
+
+This is the `Event` dataset.
+
+#### Example
+
+{{fields "event"}}

packages/microsoft_defender_cloud/changelog.yml

@@ -0,0 +1,6 @@
+# newer versions go on top
+- version: "0.1.0"
+  changes:
+    - description: Initial release.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/6593

packages/microsoft_defender_cloud/data_stream/event/_dev/test/pipeline/test-alert.log

@@ -0,0 +1,5 @@
+{"securityEventDataEnrichment":{"action":"Write","apiVersion":"2019-01-01-preview","isSnapshot":false,"interval":"00:00:00"},"id":"/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/providers/Microsoft.Security/regulatoryComplianceStandards/Microsoft-cloud-security-benchmark/regulatoryComplianceControls/LT.5/regulatoryComplianceAssessments/45cfe080-ceb1-a91e-9743-71551ed24e94","name":"45cfe080-ceb1-a91e-9743-71551ed24e94","type":"Microsoft.Security/regulatoryComplianceStandards/regulatoryComplianceControls/regulatoryComplianceAssessments","properties":{"description":"Log Analytics agent should be installed on virtual machine scale sets","state":"Skipped","passedResources":0,"failedResources":0,"skippedResources":1,"assessmentType":"AssessmentResult","assessmentDetailsLink":"https://portal.azure.com/#blade/Microsoft_Azure_Security/RecommendationsBlade/assessmentKey/45cfe080-ceb1-a91e-9743-71551ed24e94/initiativeId/Microsoft-cloud-security-benchmark"}}
+{"assessmentEventDataEnrichment":{"action":"Delete","apiVersion":"2019-01-01","isSnapshot":false},"securityEventDataEnrichment":{"action":"Delete","apiVersion":"2019-01-01","isSnapshot":false},"tenantId":"aa40685b-417d-4664-b4ec-8f7640719adb","type":"Microsoft.Security/assessments","kind":null,"location":null,"id":"/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourcegroups/mbranca-esf/providers/microsoft.web/sites/mbranca-esf/providers/Microsoft.Security/assessments/7b3d4796-9400-2904-692b-4a5ede7f0a1e","name":"7b3d4796-9400-2904-692b-4a5ede7f0a1e","tags":null,"properties":{"resourceDetails":{"source":"Azure","id":"/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourcegroups/mbranca-esf/providers/microsoft.web/sites/mbranca-esf"},"displayName":"CORS should not allow every resource to access Function Apps","status":{"code":"Healthy","statusChangeDate":"2023-05-09T13:19:49.3381028Z","firstEvaluationDate":"2023-05-09T13:19:49.3381028Z"},"additionalData":{"kind":"Functionapp"},"metadata":{"displayName":"CORS should not allow every resource to access Function Apps","assessmentType":"BuiltIn","policyDefinitionId":"/providers/microsoft.authorization/policydefinitions/0820b7b9-23aa-4725-a1ce-ae4558f718e5","description":"Cross-Origin Resource Sharing (CORS) should not allow all domains to access your Function app. Allow only required domains to interact with your Function app.","remediationDescription":"To allow only required domains to interact with your web app, we recommend the following steps:<br>1. Go to the app service CORS page<br>2. Remove the \"*\" defined and instead specify explicit origins that should be allowed to make cross-origin calls<br>3. Click Save","categories":["AppServices"],"severity":"Low","userImpact":"Low","implementationEffort":"Low","threats":["MaliciousInsider","AccountBreach"]},"links":{"azurePortal":"portal.azure.com/#blade/Microsoft_Azure_Security/RecommendationsBlade/assessmentKey/7b3d4796-9400-2904-692b-4a5ede7f0a1e/resourceId/%2fsubscriptions%2f12cabcb4-86e8-404f-a3d2-1dc9982f45ca%2fresourcegroups%2fmbranca-esf%2fproviders%2fmicrosoft.web%2fsites%2fmbranca-esf"}}}
+{"securityEventDataEnrichment":{"action":"Insert","apiVersion":"2020-01-01","isSnapshot":false},"id":"/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/providers/Microsoft.Security/secureScores/ascScore/secureScoreControls/61702b76-1fab-41f2-bcbc-50b7870dcf38","name":"61702b76-1fab-41f2-bcbc-50b7870dcf38","type":"Microsoft.Security/secureScores/secureScoreControls","properties":{"displayName":"Apply system updates","healthyResourceCount":0,"unhealthyResourceCount":3,"notApplicableResourceCount":1,"score":{"max":6,"current":0,"percentage":0},"definition":{"id":"/providers/Microsoft.Security/secureScoreControlDefinitions/61702b76-1fab-41f2-bcbc-50b7870dcf38","name":"61702b76-1fab-41f2-bcbc-50b7870dcf38","type":"Microsoft.Security/secureScoreControlDefinitions","properties":{"source":{"sourceType":"BuiltIn"},"displayName":"Apply system updates","maxScore":6,"assessmentDefinitions":[{"id":"/providers/Microsoft.Security/assessmentMetadata/d1db3318-01ff-16de-29eb-28b344515626"},{"id":"/providers/Microsoft.Security/assessmentMetadata/27ac71b1-75c5-41c2-adc2-858f5db45b08"},{"id":"/providers/Microsoft.Security/assessmentMetadata/720a3e77-0b9a-4fa9-98b6-ddf0fd7e32c1"},{"id":"/providers/Microsoft.Security/assessmentMetadata/4ab6e3c5-74dd-8b35-9ab9-f61b30875b27"},{"id":"/providers/Microsoft.Security/assessmentMetadata/e1145ab1-eb4f-43d8-911b-36ddf771d13f"},{"id":"/providers/Microsoft.Security/assessmentMetadata/90386950-71ca-4357-a12e-486d1679427c"},{"id":"/providers/Microsoft.Security/assessmentMetadata/45cfe080-ceb1-a91e-9743-71551ed24e94"},{"id":"/providers/Microsoft.Security/assessmentMetadata/bd20bd91-aaf1-7f14-b6e4-866de2f43146"},{"id":"/providers/Microsoft.Security/assessmentMetadata/bc85a7ee-7f43-47ab-8736-4faaec9346b5"},{"id":"/providers/Microsoft.Security/assessmentMetadata/11c3f3c8-3c13-48be-9ee5-67b6865e7462"},{"id":"/providers/Microsoft.Security/assessmentMetadata/643a00cb-3da3-43ef-b523-15a0f3198e45"},{"id":"/providers/Microsoft.Security/assessmentMetadata/d352afac-cebc-4e02-b474-7ef402fb1d65"}]}},"weight":3}}
+{"$type":"subAssessmentEvent","SubAssessmentEventDataEnrichment":{"$type":"subAssessmentEventDataEnrichment","Action":"Delete","ApiVersion":"2020-01-01","IsSnapshot":false},"SecurityEventDataEnrichment":{"$type":"subAssessmentEventDataEnrichment","Action":"Delete","ApiVersion":"2020-01-01","IsSnapshot":false},"TenantId":"aa40685b-417d-4664-b4ec-8f7640719adb","Type":"Microsoft.Security/assessments/subAssessments","Id":"/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourcegroups/mbranca-sdh-3372/providers/microsoft.compute/virtualmachines/sdh-3372/providers/Microsoft.Security/assessments/c476dc48-8110-4139-91af-c8d940896b98/subassessments/93d2736e-7329-8806-3ef6-e71bb2203d11","Name":"93d2736e-7329-8806-3ef6-e71bb2203d11","Properties":{"$type":"response/subAssessmentProperties","Id":"93d2736e-7329-8806-3ef6-e71bb2203d11","DisplayName":"Ensure DCCP is disabled","Status":{"$type":"status","Code":"Unhealthy","Severity":"Low"},"Remediation":"Edit or create a file in the `/etc/modprobe.d/` directory ending in .conf and add `install dccp /bin/true` then unload the dccp module or run '/opt/microsoft/omsagent/plugin/omsremediate -r disable-unnecessary-kernel-mods'","Impact":"If the protocol is not required, it is recommended that the drivers not be installed to reduce the potential attack surface.","Category":"N/A","Description":"Ensure DCCP is disabled","TimeGenerated":"2023-05-12T09:58:32.2607101Z","ResourceDetails":{"$type":"resourceDetails/azure","Source":"Azure","Id":"/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourcegroups/mbranca-sdh-3372/providers/microsoft.compute/virtualmachines/sdh-3372"},"AdditionalData":{"$type":"additionalData/general","AssessedResourceType":"GeneralVulnerability","Data":{"OsName":"Linux","RuleType":"Command","Vulnerability":"","AZID":"MSID 54","DataSourceType":"Not Applicable","DataSourceKey":"Not Applicable"}}}}
+{"VendorName":"Microsoft","AlertType":"ARM_AnomalousServiceOperation.CredentialAccess","ProductName":"Microsoft Defender for Cloud","StartTimeUtc":"2023-05-11T13:15:45.0170422Z","EndTimeUtc":"2023-05-11T13:15:45.0170422Z","TimeGenerated":"2023-05-11T13:17:09.0170422Z","ProcessingEndTime":"2023-05-11T13:17:09.0170422Z","Severity":"Medium","Status":"New","ProviderAlertStatus":null,"ConfidenceLevel":null,"ConfidenceScore":null,"ConfidenceReasons":null,"IsIncident":false,"SystemAlertId":"2517184898549829577_cdcf9f94-ec53-47a6-ab87-76130f87218d","CorrelationKey":null,"Intent":"PreAttack","AzureResourceId":"/SUBSCRIPTIONS/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/RESOURCEGROUPS/Sample-RG/providers/Microsoft.Compute/virtualMachines/Sample-VM","WorkspaceId":"00000000-0000-0000-0000-000000000001","WorkspaceSubscriptionId":"00000000-0000-0000-0000-000000000001","WorkspaceResourceGroup":"Sample-RG","AgentId":null,"CompromisedEntity":"Sample-VM","AlertDisplayName":"[SAMPLE ALERT] Login from a suspicious IP","Description":"THIS IS A SAMPLE ALERT: Your resource has been accessed successfully from an IP address that Microsoft Threat Intelligence has associated with suspicious activity.","Entities":[{"$id":"5","Address":"81.2.69.142","Location":{"CountryCode":"US","CountryName":"United States","State":"Virginia","City":"Washington","Longitude":-78.17197,"Latitude":38.73078,"Asn":8075},"ThreatIntelligence":[{"ProviderName":"AlertSimulator","ThreatType":"Sample-Type","ThreatName":"Sample-Threat","Confidence":1,"ThreatDescription":""}],"Asset":false,"Type":"ip"},{"$id":"6","ImageId":"sample-image:v1","Asset":false,"Type":"container-image"},{"$ref":"6"},{"$id":"5","DnsDomain":"","NTDomain":"","HostName":"Sample-VM","NetBiosName":"Sample-VM","AzureID":"/SUBSCRIPTIONS/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/RESOURCEGROUPS/Sample-RG/providers/Microsoft.Compute/virtualMachines/Sample-VM","OMSAgentID":"00000000-0000-0000-0000-000000000001","OSFamily":"Linux","OSVersion":"Linux","Asset":false,"Type":"host"},{"$id":"6","ProcessId":"0x1e49a","CommandLine":"","Host":{"$ref":"5"},"Asset":false,"Type":"process"},{"$id":"7","Name":"Sample-account","Host":{"$ref":"5"},"Sid":"","Asset":false,"Type":"account","LogonId":"0xbd6e"},{"$id":"9","ProcessId":"0x1e99b","CommandLine":"php","CreationTimeUtc":"2023-05-11T13:17:49.1333596Z","ImageFile":{"$ref":"8"},"Account":{"$ref":"7"},"ParentProcess":{"$ref":"6"},"Host":{"$ref":"5"},"Asset":false,"Type":"process"},{"$id":"5","DomainName":"sample.domain","IpAddresses":[{"$id":"6","Address":"81.2.69.142","Location":{"CountryCode":"US","CountryName":"United States","State":"California","City":"San Francisco","Longitude":0,"Latitude":0,"Asn":0},"Asset":false,"Type":"ip"}],"HostIpAddress":{"$ref":"6"},"Asset":false,"Type":"dns"},{"$id":"6","Address":"81.2.69.142","Location":{"CountryCode":"sample","CountryName":"united states","State":"texas","City":"san antonio","Longitude":0,"Latitude":0,"Asn":0,"Carrier":"sample","Organization":"sample-organization","OrganizationType":"sample-organization","CloudProvider":"Azure","SystemService":"sample"},"ThreatIntelligence":[{"ProviderName":"Sample-Provider","ThreatType":"Sample-Threat","ThreatName":"Sample-Threat","Confidence":0.8,"ThreatDescription":"Sample-Threat"}],"Asset":false,"Type":"ip"},{"$id":"5","HostName":"Sample-VM","AzureID":"/SUBSCRIPTIONS/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/RESOURCEGROUPS/Sample-RG/providers/Microsoft.Compute/virtualMachines/Sample-VM","OMSAgentID":"00000000-0000-0000-0000-000000000000","Asset":false,"Type":"host"},{"$id":"7","Directory":"Sample-fileShare/dummy/path/to","Name":"Sample-Name","FileHashes":[{"$id":"8","Algorithm":"MD5","Value":"Sample-SHA","Asset":false,"Type":"filehash"}],"Asset":false,"Type":"file"},{"$id":"9","Name":"Sample-Name","Category":"Virus","Files":[{"$ref":"8"}],"Asset":false,"Type":"malware"},{"$id":"5","DomainName":"sample.domain","IpAddresses":[{"$id":"6","Address":"81.2.69.142","Location":{"CountryCode":"US","CountryName":"United States","State":"California","City":"San Francisco","Longitude":0,"Latitude":0,"Asn":0},"Asset":false,"Type":"ip"}],"HostIpAddress":{"$ref":"6"},"Asset":false,"Type":"dns"},{"$id":"7","Name":"Sample-account","NTDomain":"Sample-VM","Host":{"$ref":"5"},"Sid":"S-1-5-21-3061399664-1673012318-3185014992-20022","IsDomainJoined":false,"Asset":false,"Type":"account","LogonId":"0x427d8dd9"},{"$id":"7","Name":"Sample-namespace","Cluster":{"$ref":"5"},"Asset":false,"Type":"K8s-namespace"},{"$id":"8","Name":"sample-pod","Namespace":{"$ref":"7"},"Asset":false,"Type":"K8s-pod"},{"$id":"9","Name":"sample-container","Image":{"$ref":"4"},"Pod":{"$ref":"8"},"Asset":false,"Type":"container"},{"$id":"10","ProjectId":"012345678901","ResourceType":"GCP Resource","ResourceName":"Sample-Cluster","Location":"us-central1-c","LocationType":"Zonal","Asset":true,"Type":"gcp-resource","RelatedAzureResourceIds":["/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/Sample-RG/providers/Microsoft.Security/securityConnectors/gcp-connector/securityentitydata/gcp-clusters-sample-cluster-us-central1-c"]},{"$id":"7","Name":"Sample-Name","BlobContainer":{"$ref":"5"},"Url":"https://Sample-Storage.blob.core.windows.net/Sample/Sample.txt","Etag":"Sample-Tag","Asset":false,"Type":"blob"},{"$id":"5","Name":"sample","UPNSuffix":"contoso.com","AadTenantId":"00000000-0000-0000-0000-000000000000","AadUserId":"00000000-0000-0000-0000-000000000000","Asset":false,"Type":"account"},{"$id":"5","CloudResource":{"$ref":"4"},"Asset":false,"Type":"K8s-cluster"},{"$id":"8","Directory":"https://Sample-Storage.blob.core.windows.net/Sample","Name":"Sample-Name","FileHashes":[{"$ref":"6"}],"Asset":false,"Type":"file"},{"$id":"10","ProjectId":"012345678901","ResourceType":"GCP Resource","ResourceName":"Sample-Cluster","Location":"us-central1-c","LocationType":"Zonal","Asset":true,"Type":"gcp-resource","RelatedAzureResourceIds":["/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/Sample-RG/providers/Microsoft.Security/securityConnectors/gcp-connector/securityentitydata/gcp-clusters-sample-cluster-us-central1-c"]},{"$id":"6","SourceAddress":{"$ref":"5"},"Protocol":"Tcp","Asset":false,"Type":"network-connection"},{"$id":"7","Name":"Sample-Name","StorageResource":{"$ref":"4"},"Asset":false,"Type":"blob-container"},{"$id":"7","ContainerId":"cc8ec8580f4c","Image":{"$ref":"6"},"Asset":false,"Type":"container"},{"$id":"5","Address":"81.2.69.142","Location":{"CountryCode":"IN","CountryName":"United States","State":"Virginia","City":"Washington","Longitude":-78.17197,"Latitude":38.73078,"Asn":8075},"ThreatIntelligence":[{"ProviderName":"AlertSimulator","ThreatType":"Sample-Type","ThreatName":"Sample-Threat","Confidence":1,"ThreatDescription":""}],"Asset":false,"Type":"ip"}],"ExtendedLinks":[{"Href":"https://blog.netspi.com/gathering-bearer-tokens-azure/","Category":null,"Label":"NetSPI blogpost","Type":"webLink"},{"Href":"https://github.com/NetSPI/MicroBurst/blob/master/REST/Get-AZStorageKeysREST.ps1","Category":null,"Label":"MicroBurst source code","Type":"webLink"}],"ResourceIdentifiers":[{"$id":"2","AzureResourceId":"/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca","Type":"AzureResource","AzureResourceTenantId":"aa40685b-417d-4664-b4ec-8f7640719adb"},{"$id":"3","AadTenantId":"aa40685b-417d-4664-b4ec-8f7640719adb","Type":"AAD"},{"$id":"3","WorkspaceId":"00000000-0000-0000-0000-000000000001","WorkspaceSubscriptionId":"00000000-0000-0000-0000-000000000001","WorkspaceResourceGroup":"Sample-RG","AgentId":"00000000-0000-0000-0000-00000000000","Type":"LogAnalytics"}],"RemediationSteps":["Go to the firewall settings in order to lock down the firewall as tightly as possible."],"ExtendedProperties":{"resourceType":"Virtual Machine","Investigation steps":"{\"displayValue\":\"How to investigate this alert using logs at your Log Analytics workspace.\",\"kind\":\"Link\",\"value\":\"https:\\/\\/go.microsoft.com\\/fwlink\\/?linkid=2091064\"}","Potential causes":"An attacker has accessed your database from a potentially suspicious IP; a legitimate user has accessed your database from a potentially suspicious IP.","Client principal name":"Sample-user","Alert Id":"00000000-0000-0000-0000-000000000000","Client IP address":"81.2.69.142","Client IP location":"Sample","Client application":"Sample-app","OMS workspace ID":"00000000-0000-0000-0000-000000000001","OMS agent ID":"00000000-0000-0000-0000-000000000001"},"AlertUri":"https://portal.azure.com/#blade/Microsoft_Azure_Security_AzureDefenderForData/AlertBlade/alertId/2517184898549829577_cdcf9f94-ec53-47a6-ab87-76130f87218d/subscriptionId/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroup/Sample-RG/referencedFrom/alertDeepLink/location/centralus"}