[vSphere][datastore] Add support for additional metrics #10911

lucian-ioan · 2024-08-28T09:02:13Z

Proposed commit message

Following beats code changes in elastic/beats#40441, the following metrics need integration update:

Metrics Type	Metrics	Field Mappings
Datastore performance
	datastore.read.average (KBps)	datastore.read.bytes
	datastore.write.average (KBps)	datastore.write.bytes
	disk.capacity.bytes (KBps)	disk.capacity.bytes
	disk.capacity.usage.average (KB)	disk.capacity.usage.bytes
	disk.provisioned.bytes (KBps)	disk.provisioned.bytes
Datastore Summary
	Datastore.OverallStatus	datastore.status
	Datastore.Host	datastore.host.count
	Datastore.Host	datastore.host.names
	Datastore.Vm	datastore.vm.count
	Datastore.Vm	datastore.vm.names

Alerts
triggerd_alarms.name	Name of the alarm.
triggerd_alarms.id	Unique key that identifies the alarm.
triggerd_alarms.status	Overall status of the alarm object.
triggerd_alarms.triggered_time	Time the alarm triggered.
triggerd_alarms.description	Description of the alarm.
triggerd_alarms.entity_name	alarm happened in respective entity

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Create Snapshot of Metricbeat changes.
Use Snapshot to make the changes in the vSphere Integration Host datastream.
Update the system and pipeline changes.
Add versioning to changelog.
Test the Integration for metric collection.
Test Integration from TSDB Perspective.
Code review.

How to test this PR locally

Related issues

Closes https://github.com/elastic/obs-integration-team/issues/29

elasticmachine · 2024-08-28T12:17:02Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

packages/vsphere/changelog.yml

lalit-satapathy · 2024-08-30T10:57:30Z

What about stack version dependancy?

Also release version we need to use next.

ishleenk17 · 2024-09-02T05:35:08Z

I have shared the details of how we can handle versioning here.
Please follow that for all the Integration PR's

ishleenk17 · 2024-09-02T07:03:06Z

@agithomas : Keeping TSDB in mind, can you check and suggest if any of these need to be added for TSDB?

packages/vsphere/data_stream/datastore/fields/fields.yml

packages/vsphere/manifest.yml

lucian-ioan · 2024-09-02T09:59:06Z

@agithomas @ishleenk17 I think host.names and vm.names could be used for TSDB.

Do you think using the fingerprint processor is unnecessary here?

packages/vsphere/data_stream/datastore/fields/fields.yml

agithomas · 2024-09-02T13:22:39Z

@agithomas @ishleenk17 I think host.names and vm.names could be used for TSDB.

Do you think using the fingerprint processor is unnecessary here?

Going with the description of host.names, it is List of all the host names associated with the datastore.. So, can there be more than one hosts mapped to one datastore? So, is this an array?

ishleenk17 · 2024-09-02T13:42:41Z

@agithomas @ishleenk17 I think host.names and vm.names could be used for TSDB.
Do you think using the fingerprint processor is unnecessary here?

Going with the description of host.names, it is List of all the host names associated with the datastore.. So, can there be more than one hosts mapped to one datastore? So, is this an array? Does dimension mapping support arrays?

Yes, one datastore can have multiple hosts associated.

agithomas · 2024-09-02T14:01:35Z

@ishleenk17 , if a specific datastore is mapped with more than one hosts, will it create multiple documents or it will be a single document having host.names having multiple values?

This decides - if host.names must be a dimension or not

ishleenk17 · 2024-09-02T14:03:49Z

@ishleenk17 , if a specific datastore is mapped with more than one hosts, will it create multiple documents or it will be a single document having host.names having multiple values?

This decides - if host.names must be a dimension or not

Single document with host.name/vm.name as an array in it. Sample json

lucian-ioan · 2024-09-18T09:07:38Z

Updated the integration with all the recent changes from elastic/beats.

Will rerun system tests, but I don't think any other changes are needed.

ishleenk17 · 2024-09-19T07:26:21Z

Period is not removed from the main manifest file and upgrade the kibana version to 8.15.2

ishleenk17 · 2024-09-19T07:24:10Z

packages/vsphere/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.14.0-next"


Update the version to 1.15.0-next

ishleenk17 · 2024-09-19T07:24:58Z

packages/vsphere/data_stream/datastore/fields/fields.yml

+      description: >
+        Rate of writing data to the datastore.
+      format: bytes
+    - name: write.latency.total.ms


I don't think this is resolved still ?

ishleenk17 · 2024-09-19T10:33:48Z

packages/vsphere/data_stream/datastore/manifest.yml

+        title: Period
+        description: The vSphere performance API allows for collecting host and datastore metrics at various intervals, including real-time (every 20 seconds), 5 minutes, or longer durations. Properly configuring the "period" parameter is crucial to manage usage restrictions and ensure accurate data collection. For more information on optimizing your data collection setup, refer to the [Data Collection Intervals](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html) and [Data Collection Levels](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html) documentation.
+        default: 20s
+      - name: tags


Tags is not added to hbs file.

ishleenk17 · 2024-09-19T10:36:25Z

Can we add pipeline tests as well ?
You may refer to host PR

…ntegration

ishleenk17 · 2024-09-19T10:55:45Z

The fields in the commit description are not updated to the latest ones

elastic-sonarqube · 2024-09-19T11:16:02Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-09-19T11:16:28Z

💚 Build Succeeded

Buildkite Build
Commit: 9c15224

History

💔 Build #16145 failed 47317a7
💔 Build #16143 failed 1bdac8d
💚 Build #16048 succeeded 7cad138
💚 Build #16038 succeeded 7bb821b
💔 Build #16030 failed 6a47c43
💚 Build #15413 succeeded 73366a7

cc @lucian-ioan

niraj-elastic

LGTM!

kush-elastic

LGTM!

ishleenk17

Looks good!

elastic-vault-github-plugin-prod · 2024-09-26T14:30:36Z

Package vsphere - 1.15.0 containing this change is available at https://epr.elastic.co/search?package=vsphere

lucian-ioan added 4 commits August 28, 2024 11:44

update event

2004252

update fields.yml

a2c2591

add changelog entry

d81254b

update sample event

e9147ab

lucian-ioan self-assigned this Aug 28, 2024

lucian-ioan added the Integration:vsphere VMware vSphere label Aug 28, 2024

lucian-ioan marked this pull request as ready for review August 28, 2024 09:08

lucian-ioan requested a review from a team as a code owner August 28, 2024 09:08

elastic-package build

2eb74a8

andrewkroh added enhancement New feature or request Team:Obs-InfraObs Label for the Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Aug 28, 2024

lucian-ioan added 2 commits August 28, 2024 14:34

fix sample event

11a135b

fix vsphere name and fstype

8ce790b

ishleenk17 reviewed Aug 29, 2024

View reviewed changes

packages/vsphere/changelog.yml Outdated Show resolved Hide resolved

ishleenk17 reviewed Sep 2, 2024

View reviewed changes

lucian-ioan added 3 commits September 2, 2024 12:53

follow changelog guidelines

321a5e2

update fields

0cbd1bc

better field descriptions

0677108

lucian-ioan added 2 commits September 2, 2024 13:06

update sample event

3b98122

package build

38b42aa

agithomas reviewed Sep 2, 2024

View reviewed changes

packages/vsphere/data_stream/datastore/fields/fields.yml Outdated Show resolved Hide resolved

lucian-ioan force-pushed the update_vsphere_datastore_integration branch from ea12d0d to 57aad0b Compare September 18, 2024 08:30

lucian-ioan and others added 3 commits September 18, 2024 11:30

Merge branch 'main' into update_vsphere_datastore_integration

4c5186f

add disk capacity fields

4c457aa

add metric types

7bb821b

lucian-ioan requested a review from ishleenk17 September 18, 2024 09:07

update tag

7cad138

ishleenk17 reviewed Sep 19, 2024

View reviewed changes

lucian-ioan added 4 commits September 19, 2024 13:10

update version to 1.15.0-next

5333a11

update kibana version to 8.15.2

73a1b47

remove global period

4afc186

remove latency and iops fields

ac62604

lucian-ioan requested a review from ishleenk17 September 19, 2024 10:20

update triggered alarms to group

9c7ae05

ishleenk17 reviewed Sep 19, 2024

View reviewed changes

lucian-ioan added 4 commits September 19, 2024 13:39

add tags in stream.yml.hbs

1bdac8d

add system tests

47317a7

Merge remote-tracking branch 'origin' into update_vsphere_datastore_i…

0b87f5b

…ntegration

update readme

9c15224

lucian-ioan requested a review from ishleenk17 September 19, 2024 11:06

niraj-elastic approved these changes Sep 19, 2024

View reviewed changes

kush-elastic approved these changes Sep 19, 2024

View reviewed changes

ishleenk17 approved these changes Sep 19, 2024

View reviewed changes

lucian-ioan merged commit 3390921 into elastic:main Sep 19, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[vSphere][datastore] Add support for additional metrics #10911

[vSphere][datastore] Add support for additional metrics #10911

lucian-ioan commented Aug 28, 2024 •

edited

Loading

elasticmachine commented Aug 28, 2024

lalit-satapathy commented Aug 30, 2024 •

edited

Loading

ishleenk17 commented Sep 2, 2024

ishleenk17 commented Sep 2, 2024

lucian-ioan commented Sep 2, 2024 •

edited

Loading

agithomas commented Sep 2, 2024 •

edited

Loading

ishleenk17 commented Sep 2, 2024

agithomas commented Sep 2, 2024

ishleenk17 commented Sep 2, 2024

lucian-ioan commented Sep 18, 2024 •

edited

Loading

ishleenk17 commented Sep 19, 2024

ishleenk17 Sep 19, 2024

ishleenk17 Sep 19, 2024

ishleenk17 Sep 19, 2024

ishleenk17 commented Sep 19, 2024

ishleenk17 commented Sep 19, 2024

elastic-sonarqube bot commented Sep 19, 2024

elasticmachine commented Sep 19, 2024

niraj-elastic left a comment

kush-elastic left a comment

ishleenk17 left a comment

elastic-vault-github-plugin-prod bot commented Sep 26, 2024

		@@ -1,4 +1,9 @@
		# newer versions go on top
		- version: "1.14.0-next"

[vSphere][datastore] Add support for additional metrics #10911

[vSphere][datastore] Add support for additional metrics #10911

Conversation

lucian-ioan commented Aug 28, 2024 • edited Loading

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

elasticmachine commented Aug 28, 2024

🚀 Benchmarks report

lalit-satapathy commented Aug 30, 2024 • edited Loading

ishleenk17 commented Sep 2, 2024

ishleenk17 commented Sep 2, 2024

lucian-ioan commented Sep 2, 2024 • edited Loading

agithomas commented Sep 2, 2024 • edited Loading

ishleenk17 commented Sep 2, 2024

agithomas commented Sep 2, 2024

ishleenk17 commented Sep 2, 2024

lucian-ioan commented Sep 18, 2024 • edited Loading

ishleenk17 commented Sep 19, 2024

ishleenk17 Sep 19, 2024

Choose a reason for hiding this comment

ishleenk17 Sep 19, 2024

Choose a reason for hiding this comment

ishleenk17 Sep 19, 2024

Choose a reason for hiding this comment

ishleenk17 commented Sep 19, 2024

ishleenk17 commented Sep 19, 2024

elastic-sonarqube bot commented Sep 19, 2024

Quality Gate passed

elasticmachine commented Sep 19, 2024

💚 Build Succeeded

History

niraj-elastic left a comment

Choose a reason for hiding this comment

kush-elastic left a comment

Choose a reason for hiding this comment

ishleenk17 left a comment

Choose a reason for hiding this comment

elastic-vault-github-plugin-prod bot commented Sep 26, 2024

lucian-ioan commented Aug 28, 2024 •

edited

Loading

lalit-satapathy commented Aug 30, 2024 •

edited

Loading

lucian-ioan commented Sep 2, 2024 •

edited

Loading

agithomas commented Sep 2, 2024 •

edited

Loading

lucian-ioan commented Sep 18, 2024 •

edited

Loading