Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[vSphere][datastore] Add support for additional metrics #10911

Merged
merged 30 commits into from
Sep 19, 2024
Merged

[vSphere][datastore] Add support for additional metrics #10911

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
2004252
update event
lucian-ioan Aug 28, 2024
a2c2591
update fields.yml
lucian-ioan Aug 28, 2024
d81254b
add changelog entry
lucian-ioan Aug 28, 2024
e9147ab
update sample event
lucian-ioan Aug 28, 2024
2eb74a8
elastic-package build
lucian-ioan Aug 28, 2024
11a135b
fix sample event
lucian-ioan Aug 28, 2024
8ce790b
fix vsphere name and fstype
lucian-ioan Aug 28, 2024
321a5e2
follow changelog guidelines
lucian-ioan Sep 2, 2024
0cbd1bc
update fields
lucian-ioan Sep 2, 2024
0677108
better field descriptions
lucian-ioan Sep 2, 2024
3b98122
update sample event
lucian-ioan Sep 2, 2024
38b42aa
package build
lucian-ioan Sep 2, 2024
2cb93db
add metric types
lucian-ioan Sep 3, 2024
9f6b19f
revert dockerfile
lucian-ioan Sep 3, 2024
73366a7
add dockerfile newline
lucian-ioan Sep 3, 2024
6a47c43
add period
lucian-ioan Sep 18, 2024
57aad0b
add triggered alarms
lucian-ioan Sep 18, 2024
4c5186f
Merge branch 'main' into update_vsphere_datastore_integration
lucian-ioan Sep 18, 2024
4c457aa
add disk capacity fields
lucian-ioan Sep 18, 2024
7bb821b
add metric types
lucian-ioan Sep 18, 2024
7cad138
update tag
lucian-ioan Sep 18, 2024
5333a11
update version to 1.15.0-next
lucian-ioan Sep 19, 2024
73a1b47
update kibana version to 8.15.2
lucian-ioan Sep 19, 2024
4afc186
remove global period
lucian-ioan Sep 19, 2024
ac62604
remove latency and iops fields
lucian-ioan Sep 19, 2024
9c7ae05
update triggered alarms to group
lucian-ioan Sep 19, 2024
1bdac8d
add tags in stream.yml.hbs
lucian-ioan Sep 19, 2024
47317a7
add system tests
lucian-ioan Sep 19, 2024
0b87f5b
Merge remote-tracking branch 'origin' into update_vsphere_datastore_i…
lucian-ioan Sep 19, 2024
9c15224
update readme
lucian-ioan Sep 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/vsphere/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.14.0-next"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update the version to 1.15.0-next

changes:
- description: Add additional metrics for datastore datastream
type: enhancement
link: https://github.com/elastic/integrations/pull/10911
- version: "1.13.0"
changes:
- description: ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.
Expand Down
65 changes: 59 additions & 6 deletions packages/vsphere/data_stream/datastore/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,33 +6,86 @@
type: keyword
# Reason to add as a dimension field: to uniquely identify the datastore
dimension: true
description: Datastore name
description: Datastore name.
- name: fstype
type: keyword
# Reason to add as a dimension field: in case datastore name is duplicated across different type.
dimension: true
description: Filesystem type
description: Filesystem type.
- name: capacity.total.bytes
type: long
metric_type: gauge
unit: byte
description: Total bytes of the datastore
description: Total bytes of the datastore.
format: bytes
- name: capacity.free.bytes
type: long
metric_type: gauge
unit: byte
description: Free bytes of the datastore
description: Free bytes of the datastore.
format: bytes
- name: capacity.used.bytes
type: long
metric_type: gauge
unit: byte
description: Used bytes of the datastore
description: Used bytes of the datastore.
format: bytes
- name: capacity.used.pct
type: scaled_float
metric_type: gauge
unit: percent
description: Used percent of the datastore
description: Percentage of datastore capacity used.
format: percent
- name: host.count
type: long
metric_type: gauge
description: >
Number of hosts associated with the datastore.
- name: host.names
agithomas marked this conversation as resolved.
Show resolved Hide resolved
type: keyword
description: >
List of all the host names associated with the datastore.
- name: iops
type: long
metric_type: gauge
description: >
Storage I/O Control aggregated Input/Output Operations Per Second.
- name: read.bytes
type: long
metric_type: gauge
unit: byte
description: >
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

metric_type to be added for new metrics.

Rate of reading data from the datastore.
format: bytes
- name: read.latency.total.ms
type: long
metric_type: gauge
unit: ms
description: >
Average amount of time for a read operation from the datastore in milliseconds.
- name: status
type: keyword
description: >
Status of the datastore.
- name: vm.count
type: long
metric_type: gauge
description: >
Number of VMs associated with the datastore.
- name: vm.names
type: keyword
description: >
List of all the VM names associated with the datastore.
- name: write.bytes
type: long
metric_type: gauge
unit: byte
description: >
Rate of writing data to the datastore.
format: bytes
- name: write.latency.total.ms
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the metricset of datastore. There is a change in the fields.
Lets get the latest fields in.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is resolved still ?

metric_type: gauge
unit: ms
type: long
description: >
Average amount of time for a write operation from the datastore in milliseconds.
72 changes: 47 additions & 25 deletions packages/vsphere/data_stream/datastore/sample_event.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,48 +1,50 @@
{
"@timestamp": "2023-06-29T08:03:30.114Z",
"@timestamp": "2024-09-02T10:04:25.122Z",
"agent": {
"ephemeral_id": "8b019ff3-cbda-41fa-b1ff-974d482b9694",
"id": "5096d7cc-1e4b-4959-abea-7355be2913a7",
"name": "docker-fleet-agent",
"ephemeral_id": "4da294a3-ad54-47f4-92c7-544e1356a0d8",
"id": "b01ab3cf-51ad-4c4d-87bd-fc2d4aa59d8a",
"name": "elastic-agent-76236",
"type": "metricbeat",
"version": "8.8.1"
"version": "8.16.0"
},
"data_stream": {
"dataset": "vsphere.datastore",
"namespace": "ep",
"namespace": "86691",
"type": "metrics"
},
"ecs": {
"version": "8.11.0"
},
"elastic_agent": {
"id": "5096d7cc-1e4b-4959-abea-7355be2913a7",
"snapshot": false,
"version": "8.8.1"
"id": "b01ab3cf-51ad-4c4d-87bd-fc2d4aa59d8a",
"snapshot": true,
"version": "8.16.0"
},
"event": {
"agent_id_status": "verified",
"dataset": "vsphere.datastore",
"duration": 23155458,
"ingested": "2023-06-29T08:03:31Z",
"duration": 190147614,
"ingested": "2024-09-02T10:04:27Z",
"module": "vsphere"
},
"host": {
"architecture": "aarch64",
"containerized": false,
"hostname": "docker-fleet-agent",
"id": "d08b346fbb8f49f5a2bb1a477f8ceb54",
"architecture": "x86_64",
"containerized": true,
"hostname": "elastic-agent-76236",
"id": "e744630f9d4f43dc818e497d221bd0b2",
"ip": [
"172.23.0.7"
"172.18.0.4",
"172.21.0.2"
],
"mac": [
"02-42-AC-17-00-07"
"02-42-AC-12-00-04",
"02-42-AC-15-00-02"
],
"name": "docker-fleet-agent",
"name": "elastic-agent-76236",
"os": {
"codename": "focal",
"family": "debian",
"kernel": "5.10.104-linuxkit",
"kernel": "5.15.153.1-microsoft-standard-WSL2",
"name": "Ubuntu",
"platform": "ubuntu",
"type": "linux",
Expand All @@ -54,25 +56,45 @@
"period": 10000
},
"service": {
"address": "https://elastic-package-service_vsphere-metrics_1:8989/sdk",
"address": "https://svc-vsphere-metrics:8989/sdk",
"type": "vsphere"
},
"vsphere": {
"datastore": {
"capacity": {
"free": {
"bytes": 47869427712
"bytes": 10952166604800
},
"total": {
"bytes": 62725623808
"bytes": 10995116277760
},
"used": {
"bytes": 14856196096,
"pct": 0.237
"bytes": 42949672960,
"pct": 0.004
}
},
"fstype": "OTHER",
"name": "LocalDS_0"
"host": {
"count": 1,
"names": "DC0_H0"
},
"name": "LocalDS_0",
"read": {
"bytes": 1024
},
"status": "green",
"vm": {
"count": 4,
"names": [
"DC0_C0_RP0_VM0",
"DC0_C0_RP0_VM1",
"DC0_H0_VM0",
"DC0_H0_VM1"
]
},
"write": {
"bytes": 450560
}
}
}
}
94 changes: 63 additions & 31 deletions packages/vsphere/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -299,50 +299,52 @@ An example event for `datastore` looks as following:

```json
{
"@timestamp": "2023-06-29T08:03:30.114Z",
"@timestamp": "2024-09-02T10:04:25.122Z",
"agent": {
"ephemeral_id": "8b019ff3-cbda-41fa-b1ff-974d482b9694",
"id": "5096d7cc-1e4b-4959-abea-7355be2913a7",
"name": "docker-fleet-agent",
"ephemeral_id": "4da294a3-ad54-47f4-92c7-544e1356a0d8",
"id": "b01ab3cf-51ad-4c4d-87bd-fc2d4aa59d8a",
"name": "elastic-agent-76236",
"type": "metricbeat",
"version": "8.8.1"
"version": "8.16.0"
},
"data_stream": {
"dataset": "vsphere.datastore",
"namespace": "ep",
"namespace": "86691",
"type": "metrics"
},
"ecs": {
"version": "8.11.0"
},
"elastic_agent": {
"id": "5096d7cc-1e4b-4959-abea-7355be2913a7",
"snapshot": false,
"version": "8.8.1"
"id": "b01ab3cf-51ad-4c4d-87bd-fc2d4aa59d8a",
"snapshot": true,
"version": "8.16.0"
},
"event": {
"agent_id_status": "verified",
"dataset": "vsphere.datastore",
"duration": 23155458,
"ingested": "2023-06-29T08:03:31Z",
"duration": 190147614,
"ingested": "2024-09-02T10:04:27Z",
"module": "vsphere"
},
"host": {
"architecture": "aarch64",
"containerized": false,
"hostname": "docker-fleet-agent",
"id": "d08b346fbb8f49f5a2bb1a477f8ceb54",
"architecture": "x86_64",
"containerized": true,
"hostname": "elastic-agent-76236",
"id": "e744630f9d4f43dc818e497d221bd0b2",
"ip": [
"172.23.0.7"
"172.18.0.4",
"172.21.0.2"
],
"mac": [
"02-42-AC-17-00-07"
"02-42-AC-12-00-04",
"02-42-AC-15-00-02"
],
"name": "docker-fleet-agent",
"name": "elastic-agent-76236",
"os": {
"codename": "focal",
"family": "debian",
"kernel": "5.10.104-linuxkit",
"kernel": "5.15.153.1-microsoft-standard-WSL2",
"name": "Ubuntu",
"platform": "ubuntu",
"type": "linux",
Expand All @@ -354,25 +356,45 @@ An example event for `datastore` looks as following:
"period": 10000
},
"service": {
"address": "https://elastic-package-service_vsphere-metrics_1:8989/sdk",
"address": "https://svc-vsphere-metrics:8989/sdk",
"type": "vsphere"
},
"vsphere": {
"datastore": {
"capacity": {
"free": {
"bytes": 47869427712
"bytes": 10952166604800
},
"total": {
"bytes": 62725623808
"bytes": 10995116277760
},
"used": {
"bytes": 14856196096,
"pct": 0.237
"bytes": 42949672960,
"pct": 0.004
}
},
"fstype": "OTHER",
"name": "LocalDS_0"
"host": {
"count": 1,
"names": "DC0_H0"
},
"name": "LocalDS_0",
"read": {
"bytes": 1024
},
"status": "green",
"vm": {
"count": 4,
"names": [
"DC0_C0_RP0_VM0",
"DC0_C0_RP0_VM1",
"DC0_H0_VM0",
"DC0_H0_VM1"
]
},
"write": {
"bytes": 450560
}
}
}
}
Expand Down Expand Up @@ -404,12 +426,22 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
| host.os.build | OS build information. | keyword | | |
| host.os.codename | OS codename, if any. | keyword | | |
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | |
| vsphere.datastore.capacity.free.bytes | Free bytes of the datastore | long | byte | gauge |
| vsphere.datastore.capacity.total.bytes | Total bytes of the datastore | long | byte | gauge |
| vsphere.datastore.capacity.used.bytes | Used bytes of the datastore | long | byte | gauge |
| vsphere.datastore.capacity.used.pct | Used percent of the datastore | scaled_float | percent | gauge |
| vsphere.datastore.fstype | Filesystem type | keyword | | |
| vsphere.datastore.name | Datastore name | keyword | | |
| vsphere.datastore.capacity.free.bytes | Free bytes of the datastore. | long | byte | gauge |
| vsphere.datastore.capacity.total.bytes | Total bytes of the datastore. | long | byte | gauge |
| vsphere.datastore.capacity.used.bytes | Used bytes of the datastore. | long | byte | gauge |
| vsphere.datastore.capacity.used.pct | Percentage of datastore capacity used. | scaled_float | percent | gauge |
| vsphere.datastore.fstype | Filesystem type. | keyword | | |
| vsphere.datastore.host.count | Number of hosts associated with the datastore. | long | | gauge |
| vsphere.datastore.host.names | List of all the host names associated with the datastore. | keyword | | |
| vsphere.datastore.iops | Storage I/O Control aggregated Input/Output Operations Per Second. | long | | gauge |
| vsphere.datastore.name | Datastore name. | keyword | | |
| vsphere.datastore.read.bytes | Rate of reading data from the datastore. | long | byte | gauge |
| vsphere.datastore.read.latency.total.ms | Average amount of time for a read operation from the datastore in milliseconds. | long | ms | gauge |
| vsphere.datastore.status | Status of the datastore. | keyword | | |
| vsphere.datastore.vm.count | Number of VMs associated with the datastore. | long | | gauge |
| vsphere.datastore.vm.names | List of all the VM names associated with the datastore. | keyword | | |
| vsphere.datastore.write.bytes | Rate of writing data to the datastore. | long | byte | gauge |
| vsphere.datastore.write.latency.total.ms | Average amount of time for a write operation from the datastore in milliseconds. | long | ms | gauge |


## Logs
Expand Down