[O11y][Citrix ADC] Resolve the conflicts in host.ip field

packages/citrix_adc/_dev/build/docs/README.md

@@ -45,8 +45,105 @@ After the integration is successfully configured, clicking on the Assets tab of
 
 ### Troubleshooting
 
+#### Dummy values
+
 There could be a possibility that for some of the fields, Citrix ADC sets dummy values. For example, a field `cpuusagepcnt` is represented by `citrix_adc.system.cpu.utilization.pct`. `cpuusagepcnt` is set to `4294967295` for some [instances](https://github.com/citrix/citrix-adc-metrics-exporter/issues/44). If you also encounter it for some fields please reach out to the [Citrix ADC support team](https://support.citrix.com/plp/products/citrix_adc/tabs/popular-solutions).
 
+
+#### Type conflicts
+
+If host.ip is shown conflicted under ``logs-*`` data view, then this issue can be solved by reindexing the ``Interface``, ``LBVserver``, ``Service``, ``System``, and ``VPN`` data stream's indices.
+To reindex the data, the following steps must be performed.
+
+1. Stop the data stream by going to `Integrations -> Citrix ADC -> Integration policies` open the configuration of Citrix ADC and disable the `Collect Citrix ADC metrics` toggle to reindex logs data stream and save the integration.
+
+2. Copy data into the temporary index and delete the existing data stream and index template by performing the following steps in the Dev tools.
+
+```
+POST _reindex
+{
+  "source": {
+    "index": "<index_name>"
+  },
+  "dest": {
+    "index": "temp_index"
+  }
+}  
+```
+Example:
+```
+POST _reindex
+{
+  "source": {
+    "index": "logs-citrix_adc.interface-default"
+  },
+  "dest": {
+    "index": "temp_index"
+  }
+}
+```
+
+```
+DELETE /_data_stream/<data_stream>
+```
+Example:
+```
+DELETE /_data_stream/logs-citrix_adc.interface-default
+```
+
+```
+DELETE _index_template/<index_template>
+```
+Example:
+```
+DELETE _index_template/logs-citrix_adc.interface
+```
+3. Go to `Integrations -> Citrix ADC -> Settings` and click on `Reinstall Citrix ADC`.
+
+4. Copy data from temporary index to new index by performing the following steps in the Dev tools.
+
+```
+POST _reindex
+{
+  "conflicts": "proceed",
+  "source": {
+    "index": "temp_index"
+  },
+  "dest": {
+    "index": "<index_name>",
+    "op_type": "create"
+
+  }
+}
+```
+Example:
+```
+POST _reindex
+{
+  "conflicts": "proceed",
+  "source": {
+    "index": "temp_index"
+  },
+  "dest": {
+    "index": "logs-citrix_adc.interface-default",
+    "op_type": "create"
+
+  }
+}
+```
+
+5. Verify data is reindexed completely.
+
+6. Start the data stream by going to the `Integrations -> Citrix ADC -> Integration policies` and open configuration of integration and enable the `Collect Citrix ADC metrics` toggle and save the integration.
+
+7. Delete temporary index by performing the following step in the Dev tools.
+
+```
+DELETE temp_index
+```
+
+More details about reindexing can be found [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html).
+
 ## Metrics reference
 
 ### Interface

packages/citrix_adc/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "0.7.1"
+  changes:
+    - description: Resolve host.ip field conflict.
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/1 #FIX ME
 - version: "0.7.0"
   changes:
     - description: Rename ownership from obs-service-integrations to obs-infraobs-integrations

packages/citrix_adc/data_stream/interface/fields/ecs.yml

@@ -14,6 +14,8 @@
   name: event.module
 - external: ecs
   name: event.type
+- external: ecs
+  name: host.ip
 - external: ecs
   name: interface.id
 - external: ecs

packages/citrix_adc/data_stream/lbvserver/fields/ecs.yml

@@ -14,6 +14,8 @@
   name: event.module
 - external: ecs
   name: event.type
+- external: ecs
+  name: host.ip
 - external: ecs
   name: related.ip
 - external: ecs

packages/citrix_adc/data_stream/service/fields/ecs.yml

@@ -14,6 +14,8 @@
   name: event.module
 - external: ecs
   name: event.type
+- external: ecs
+  name: host.ip
 - external: ecs
   name: related.ip
 - external: ecs

packages/citrix_adc/data_stream/system/elasticsearch/ingest_pipeline/default.yml

@@ -124,10 +124,10 @@ processors:
             def bytes = (megabytes*1024*1024);
             return bytes;
         }
-        if(ctx.citrix_adc?.system?.memory?.size?.value!=null || ctx.citrix_adc?.system?.memory?.size?.value!=""){
+        if(ctx.citrix_adc?.system?.memory?.size?.value!=null && ctx.citrix_adc?.system?.memory?.size?.value!=""){
             ctx.citrix_adc.system.memory.size.value = convert(ctx.citrix_adc.system.memory.size.value);
         }
-        if(ctx.citrix_adc?.system?.memory?.usage?.value!=null || ctx.citrix_adc?.system?.memory?.usage?.value!=""){
+        if(ctx.citrix_adc?.system?.memory?.usage?.value!=null && ctx.citrix_adc?.system?.memory?.usage?.value!=""){
             ctx.citrix_adc.system.memory.usage.value = convert(ctx.citrix_adc.system.memory.usage.value);
         }
   - script:

packages/citrix_adc/data_stream/system/fields/ecs.yml

@@ -14,5 +14,7 @@
   name: event.module
 - external: ecs
   name: event.type
+- external: ecs
+  name: host.ip
 - external: ecs
   name: tags

packages/citrix_adc/data_stream/vpn/fields/ecs.yml

@@ -14,5 +14,7 @@
   name: event.module
 - external: ecs
   name: event.type
+- external: ecs
+  name: host.ip
 - external: ecs
   name: tags

packages/citrix_adc/docs/README.md

@@ -45,8 +45,105 @@ After the integration is successfully configured, clicking on the Assets tab of
 
 ### Troubleshooting
 
+#### Dummy values
+
 There could be a possibility that for some of the fields, Citrix ADC sets dummy values. For example, a field `cpuusagepcnt` is represented by `citrix_adc.system.cpu.utilization.pct`. `cpuusagepcnt` is set to `4294967295` for some [instances](https://github.com/citrix/citrix-adc-metrics-exporter/issues/44). If you also encounter it for some fields please reach out to the [Citrix ADC support team](https://support.citrix.com/plp/products/citrix_adc/tabs/popular-solutions).
 
+
+#### Type conflicts
+
+If host.ip is shown conflicted under ``logs-*`` data view, then this issue can be solved by reindexing the ``Interface``, ``LBVserver``, ``Service``, ``System``, and ``VPN`` data stream's indices.
+To reindex the data, the following steps must be performed.
+
+1. Stop the data stream by going to `Integrations -> Citrix ADC -> Integration policies` open the configuration of Citrix ADC and disable the `Collect Citrix ADC metrics` toggle to reindex logs data stream and save the integration.
+
+2. Copy data into the temporary index and delete the existing data stream and index template by performing the following steps in the Dev tools.
+
+```
+POST _reindex
+{
+  "source": {
+    "index": "<index_name>"
+  },
+  "dest": {
+    "index": "temp_index"
+  }
+}  
+```
+Example:
+```
+POST _reindex
+{
+  "source": {
+    "index": "logs-citrix_adc.interface-default"
+  },
+  "dest": {
+    "index": "temp_index"
+  }
+}
+```
+
+```
+DELETE /_data_stream/<data_stream>
+```
+Example:
+```
+DELETE /_data_stream/logs-citrix_adc.interface-default
+```
+
+```
+DELETE _index_template/<index_template>
+```
+Example:
+```
+DELETE _index_template/logs-citrix_adc.interface
+```
+3. Go to `Integrations -> Citrix ADC -> Settings` and click on `Reinstall Citrix ADC`.
+
+4. Copy data from temporary index to new index by performing the following steps in the Dev tools.
+
+```
+POST _reindex
+{
+  "conflicts": "proceed",
+  "source": {
+    "index": "temp_index"
+  },
+  "dest": {
+    "index": "<index_name>",
+    "op_type": "create"
+
+  }
+}
+```
+Example:
+```
+POST _reindex
+{
+  "conflicts": "proceed",
+  "source": {
+    "index": "temp_index"
+  },
+  "dest": {
+    "index": "logs-citrix_adc.interface-default",
+    "op_type": "create"
+
+  }
+}
+```
+
+5. Verify data is reindexed completely.
+
+6. Start the data stream by going to the `Integrations -> Citrix ADC -> Integration policies` and open configuration of integration and enable the `Collect Citrix ADC metrics` toggle and save the integration.
+
+7. Delete temporary index by performing the following step in the Dev tools.
+
+```
+DELETE temp_index
+```
+
+More details about reindexing can be found [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html).
+
 ## Metrics reference
 
 ### Interface
@@ -257,6 +354,7 @@ An example event for `interface` looks as following:
 | event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. | keyword |  |  |
 | event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | keyword |  |  |
 | event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword |  |  |
+| host.ip | Host ip addresses. | ip |  |  |
 | input.type | Type of Filebeat input. | keyword |  |  |
 | interface.id | Interface ID as reported by an observer (typically SNMP interface ID). | keyword |  |  |
 | tags | List of keywords used to tag each event. | keyword |  |  |
@@ -484,6 +582,7 @@ An example event for `lbvserver` looks as following:
 | event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. | keyword |  |  |
 | event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | keyword |  |  |
 | event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword |  |  |
+| host.ip | Host ip addresses. | ip |  |  |
 | input.type | Type of Filebeat input. | keyword |  |  |
 | related.ip | All of the IPs seen on your event. | ip |  |  |
 | server.ip | IP address of the server (IPv4 or IPv6). | ip |  |  |
@@ -654,6 +753,7 @@ An example event for `service` looks as following:
 | event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. | keyword |  |  |
 | event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | keyword |  |  |
 | event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword |  |  |
+| host.ip | Host ip addresses. | ip |  |  |
 | input.type | Type of Filebeat input. | keyword |  |  |
 | related.ip | All of the IPs seen on your event. | ip |  |  |
 | service.name | Name of the service data is collected from. The name of the service is normally user given. This allows for distributed services that run on multiple hosts to correlate the related instances based on the name. In the case of Elasticsearch the `service.name` could contain the cluster name. For Beats the `service.name` is by default a copy of the `service.type` field if no name is specified. | keyword |  |  |
@@ -799,6 +899,7 @@ An example event for `system` looks as following:
 | event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. | keyword |  |  |
 | event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | keyword |  |  |
 | event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword |  |  |
+| host.ip | Host ip addresses. | ip |  |  |
 | input.type | Type of Filebeat input. | keyword |  |  |
 | tags | List of keywords used to tag each event. | keyword |  |  |
 
@@ -1042,5 +1143,6 @@ An example event for `vpn` looks as following:
 | event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. | keyword |  |
 | event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | keyword |  |
 | event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword |  |
+| host.ip | Host ip addresses. | ip |  |
 | input.type | Type of Filebeat input. | keyword |  |
 | tags | List of keywords used to tag each event. | keyword |  |

packages/citrix_adc/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 2.0.0
 name: citrix_adc
 title: Citrix ADC
-version: "0.7.0"
+version: "0.7.1"
 description: This Elastic integration collects metrics from Citrix ADC product.
 type: integration
 categories: