[azure] Fix event.duration field conflict in log data streams (#2891)

Fix event.duration field conflict in Azure Logs.

packages/azure/changelog.yml

@@ -1,3 +1,8 @@
+- version: "1.1.4"
+  changes:
+    - description: Fix `event.duration` field mapping conflict in all Azure data streams.
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/2891
 - version: "1.1.3"
   changes:
     - description: Added the `forwarded` tag by default to all log types.

packages/azure/data_stream/activitylogs/fields/ecs.yml

@@ -36,6 +36,8 @@
   external: ecs
 - name: event.created
   external: ecs
+- name: event.duration
+  external: ecs
 - name: event.id
   external: ecs
 - name: event.ingested

packages/azure/data_stream/activitylogs/sample_event.json

@@ -15,7 +15,7 @@
         "dataset": "azure.activitylogs"
     },
     "event": {
-        "duration": "0",
+        "duration": 0,
         "ingested": "2020-10-30T20:47:48.123859400Z",
         "kind": "event",
         "action": "MICROSOFT.RESOURCES/DEPLOYMENTS/WRITE",

packages/azure/data_stream/auditlogs/fields/ecs.yml

@@ -36,6 +36,8 @@
   external: ecs
 - name: event.created
   external: ecs
+- name: event.duration
+  external: ecs
 - name: event.id
   external: ecs
 - name: event.ingested

packages/azure/data_stream/auditlogs/sample_event.json

@@ -15,7 +15,7 @@
         "dataset": "azure.auditlogs"
     },
     "event": {
-        "duration": "0",
+        "duration": 0,
         "ingested": "2020-10-30T20:47:48.123859400Z",
         "kind": "event",
         "action": "MICROSOFT.RESOURCES/DEPLOYMENTS/WRITE",

packages/azure/data_stream/eventhub/fields/ecs.yml

@@ -34,6 +34,8 @@
   external: ecs
 - name: event.created
   external: ecs
+- name: event.duration
+  external: ecs
 - name: event.id
   external: ecs
 - name: event.ingested

packages/azure/data_stream/platformlogs/fields/ecs.yml

@@ -34,6 +34,8 @@
   external: ecs
 - name: event.created
   external: ecs
+- name: event.duration
+  external: ecs
 - name: event.id
   external: ecs
 - name: event.ingested

packages/azure/data_stream/signinlogs/fields/ecs.yml

@@ -36,6 +36,8 @@
   external: ecs
 - name: event.created
   external: ecs
+- name: event.duration
+  external: ecs
 - name: event.id
   external: ecs
 - name: event.ingested

packages/azure/data_stream/springcloudlogs/fields/ecs.yml

@@ -34,6 +34,8 @@
   external: ecs
 - name: event.created
   external: ecs
+- name: event.duration
+  external: ecs
 - name: event.id
   external: ecs
 - name: event.ingested

packages/azure/docs/README.md

@@ -81,7 +81,7 @@ An example event for `activitylogs` looks as following:
         "dataset": "azure.activitylogs"
     },
     "event": {
-        "duration": "0",
+        "duration": 0,
         "ingested": "2020-10-30T20:47:48.123859400Z",
         "kind": "event",
         "action": "MICROSOFT.RESOURCES/DEPLOYMENTS/WRITE",
@@ -222,6 +222,7 @@ An example event for `activitylogs` looks as following:
 | event.category | This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories. | keyword |
 | event.created | event.created contains the date/time when the event was first read by an agent, or by your pipeline. This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event. In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source. In case the two timestamps are identical, @timestamp should be used. | date |
 | event.dataset | Event dataset | constant_keyword |
+| event.duration | Duration of the event in nanoseconds. If event.start and event.end are known this value should be the difference between the end and start time. | long |
 | event.id | Unique ID to describe the event. | keyword |
 | event.ingested | Timestamp when an event arrived in the central data store. This is different from `@timestamp`, which is when the event originally occurred.  It's also different from `event.created`, which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: `@timestamp` \< `event.created` \< `event.ingested`. | date |
 | event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. | keyword |
@@ -408,6 +409,7 @@ An example event for `platformlogs` looks as following:
 | event.category | This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories. | keyword |
 | event.created | event.created contains the date/time when the event was first read by an agent, or by your pipeline. This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event. In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source. In case the two timestamps are identical, @timestamp should be used. | date |
 | event.dataset | Event dataset | constant_keyword |
+| event.duration | Duration of the event in nanoseconds. If event.start and event.end are known this value should be the difference between the end and start time. | long |
 | event.id | Unique ID to describe the event. | keyword |
 | event.ingested | Timestamp when an event arrived in the central data store. This is different from `@timestamp`, which is when the event originally occurred.  It's also different from `event.created`, which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: `@timestamp` \< `event.created` \< `event.ingested`. | date |
 | event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. | keyword |
@@ -481,7 +483,7 @@ An example event for `auditlogs` looks as following:
         "dataset": "azure.auditlogs"
     },
     "event": {
-        "duration": "0",
+        "duration": 0,
         "ingested": "2020-10-30T20:47:48.123859400Z",
         "kind": "event",
         "action": "MICROSOFT.RESOURCES/DEPLOYMENTS/WRITE",
@@ -596,6 +598,7 @@ An example event for `auditlogs` looks as following:
 | event.category | This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories. | keyword |
 | event.created | event.created contains the date/time when the event was first read by an agent, or by your pipeline. This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event. In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source. In case the two timestamps are identical, @timestamp should be used. | date |
 | event.dataset | Event dataset | constant_keyword |
+| event.duration | Duration of the event in nanoseconds. If event.start and event.end are known this value should be the difference between the end and start time. | long |
 | event.id | Unique ID to describe the event. | keyword |
 | event.ingested | Timestamp when an event arrived in the central data store. This is different from `@timestamp`, which is when the event originally occurred.  It's also different from `event.created`, which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: `@timestamp` \< `event.created` \< `event.ingested`. | date |
 | event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. | keyword |
@@ -882,6 +885,7 @@ An example event for `signinlogs` looks as following:
 | event.category | This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories. | keyword |
 | event.created | event.created contains the date/time when the event was first read by an agent, or by your pipeline. This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event. In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source. In case the two timestamps are identical, @timestamp should be used. | date |
 | event.dataset | Event dataset | constant_keyword |
+| event.duration | Duration of the event in nanoseconds. If event.start and event.end are known this value should be the difference between the end and start time. | long |
 | event.id | Unique ID to describe the event. | keyword |
 | event.ingested | Timestamp when an event arrived in the central data store. This is different from `@timestamp`, which is when the event originally occurred.  It's also different from `event.created`, which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: `@timestamp` \< `event.created` \< `event.ingested`. | date |
 | event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. | keyword |

packages/azure/docs/activitylogs.md

@@ -70,7 +70,7 @@ An example event for `activitylogs` looks as following:
         "dataset": "azure.activitylogs"
     },
     "event": {
-        "duration": "0",
+        "duration": 0,
         "ingested": "2020-10-30T20:47:48.123859400Z",
         "kind": "event",
         "action": "MICROSOFT.RESOURCES/DEPLOYMENTS/WRITE",
@@ -211,6 +211,7 @@ An example event for `activitylogs` looks as following:
 | event.category | This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories. | keyword |
 | event.created | event.created contains the date/time when the event was first read by an agent, or by your pipeline. This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event. In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source. In case the two timestamps are identical, @timestamp should be used. | date |
 | event.dataset | Event dataset | constant_keyword |
+| event.duration | Duration of the event in nanoseconds. If event.start and event.end are known this value should be the difference between the end and start time. | long |
 | event.id | Unique ID to describe the event. | keyword |
 | event.ingested | Timestamp when an event arrived in the central data store. This is different from `@timestamp`, which is when the event originally occurred.  It's also different from `event.created`, which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: `@timestamp` \< `event.created` \< `event.ingested`. | date |
 | event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. | keyword |