Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GCP] Add dimensions for metrics data streams #8314

Merged
merged 17 commits into from
Dec 6, 2023
Merged

[GCP] Add dimensions for metrics data streams #8314

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
a1be387
add dimensions mappings
gpop63 Oct 26, 2023
19fffc4
update docs
gpop63 Oct 26, 2023
4dcf819
bump package version
gpop63 Oct 26, 2023
665434d
update dns docs
gpop63 Oct 26, 2023
ef93044
fix compute pipeline test
gpop63 Oct 26, 2023
51bb34c
change 60s to 5m storage period
gpop63 Oct 26, 2023
fabf3ae
remove metrics fingerprint
gpop63 Nov 6, 2023
82b243e
update docs
gpop63 Nov 6, 2023
8d7f158
add agent id ecs field
gpop63 Nov 6, 2023
c966ae9
update docs
gpop63 Nov 6, 2023
ca979a5
remove empty dimensions
gpop63 Nov 6, 2023
aa619d3
Merge branch 'main' into add_gcp_dimensions
gpop63 Nov 21, 2023
6adbd7a
update cloudrun docs
gpop63 Nov 21, 2023
826202e
Merge remote-tracking branch 'upstream/main' into add_gcp_dimensions
gpop63 Nov 21, 2023
a9f0760
Merge branch 'main' into add_gcp_dimensions
gpop63 Nov 22, 2023
5d3baa5
remove redundant dimensions
gpop63 Nov 26, 2023
b37f946
revert firestore
gpop63 Nov 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/gcp/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "2.32.1"
changes:
- description: Add dimensions mappings and the metrics_fingerprint field across all metrics data streams.
type: enhancement
link: https://github.com/elastic/integrations/pull/8314
- version: "2.32.0"
changes:
- description: Add new billing data stream fields.
Expand Down
5 changes: 5 additions & 0 deletions packages/gcp/data_stream/cloudrun_metrics/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
---
description: Pipeline for parsing GCP Cloud Run metrics.
processors:
- fingerprint:
fields:
- gcp.labels
target_field: gcp.labels_fingerprint
ignore_missing: true
- rename:
field: gcp.metrics.container.billable_instance_time.value
target_field: gcp.cloudrun_metrics.container.billable_instance_time
Expand Down
5 changes: 5 additions & 0 deletions packages/gcp/data_stream/cloudrun_metrics/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@
name: cloud
- external: ecs
name: cloud.account.id
dimension: true
- external: ecs
name: cloud.account.name
dimension: true
- external: ecs
name: cloud.availability_zone
- external: ecs
Expand All @@ -22,3 +24,6 @@
name: error.message
- external: ecs
name: service.type
- external: ecs
name: agent.id
dimension: true
6 changes: 6 additions & 0 deletions packages/gcp/data_stream/cloudrun_metrics/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
- name: gcp
type: group
description: >-
GCP module
fields:
- name: labels_fingerprint
dimension: true
type: keyword
description: >-
Hashed value of the labels field.
- name: labels
type: group
description: >-
Expand Down
5 changes: 5 additions & 0 deletions packages/gcp/data_stream/cloudsql_mysql/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
---
description: Pipeline for parsing GCP CloudSQL MySQL metrics.
processors:
- fingerprint:
fields:
- gcp.labels
target_field: gcp.labels_fingerprint
ignore_missing: true
- drop:
description: Drop if database is not MySQL.
if: "ctx?.gcp?.labels?.cloudsql?.name != 'mysql'"
Expand Down
5 changes: 5 additions & 0 deletions packages/gcp/data_stream/cloudsql_mysql/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@
name: cloud
- external: ecs
name: cloud.account.id
dimension: true
- external: ecs
name: cloud.account.name
dimension: true
- external: ecs
name: cloud.availability_zone
- external: ecs
Expand All @@ -22,3 +24,6 @@
name: error.message
- external: ecs
name: service.type
- external: ecs
name: agent.id
dimension: true
6 changes: 6 additions & 0 deletions packages/gcp/data_stream/cloudsql_mysql/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
- name: gcp
type: group
description: >-
GCP module
fields:
- name: labels_fingerprint
dimension: true
type: keyword
description: >-
Hashed value of the labels field.
- name: labels
type: group
description: >-
Expand Down
5 changes: 5 additions & 0 deletions packages/gcp/data_stream/cloudsql_postgresql/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
---
description: Pipeline for parsing GCP CloudSQL PostgreSQL metrics.
processors:
- fingerprint:
fields:
- gcp.labels
target_field: gcp.labels_fingerprint
ignore_missing: true
- drop:
description: Drop if database is not PostgreSQL.
if: "ctx?.gcp?.labels?.cloudsql?.name != 'postgres'"
Expand Down
5 changes: 5 additions & 0 deletions packages/gcp/data_stream/cloudsql_postgresql/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@
name: cloud
- external: ecs
name: cloud.account.id
dimension: true
- external: ecs
name: cloud.account.name
dimension: true
- external: ecs
name: cloud.availability_zone
- external: ecs
Expand All @@ -22,3 +24,6 @@
name: error.message
- external: ecs
name: service.type
- external: ecs
name: agent.id
dimension: true
6 changes: 6 additions & 0 deletions packages/gcp/data_stream/cloudsql_postgresql/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
- name: gcp
type: group
description: >-
GCP module
fields:
- name: labels_fingerprint
dimension: true
type: keyword
description: >-
Hashed value of the labels field.
- name: labels
type: group
description: >-
Expand Down
5 changes: 5 additions & 0 deletions packages/gcp/data_stream/cloudsql_sqlserver/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
---
description: Pipeline for parsing GCP CloudSQL SQL Server metrics.
processors:
- fingerprint:
fields:
- gcp.labels
target_field: gcp.labels_fingerprint
ignore_missing: true
- drop:
description: Drop if database is not SQLServer.
if: "ctx?.gcp?.labels?.cloudsql?.name != 'sqlserver'"
Expand Down
5 changes: 5 additions & 0 deletions packages/gcp/data_stream/cloudsql_sqlserver/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@
name: cloud
- external: ecs
name: cloud.account.id
dimension: true
- external: ecs
name: cloud.account.name
dimension: true
- external: ecs
name: cloud.availability_zone
- external: ecs
Expand All @@ -22,3 +24,6 @@
name: error.message
- external: ecs
name: service.type
- external: ecs
name: agent.id
dimension: true
6 changes: 6 additions & 0 deletions packages/gcp/data_stream/cloudsql_sqlserver/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
- name: gcp
type: group
description: >-
GCP module
fields:
- name: labels_fingerprint
dimension: true
type: keyword
description: >-
Hashed value of the labels field.
- name: labels
type: group
description: >-
Expand Down
3 changes: 2 additions & 1 deletion packages/gcp/data_stream/compute/_dev/test/pipeline/test-gcp-compute.json-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,8 @@
"user": {
"goog-gke-node": ""
}
}
},
"labels_fingerprint": "vP7l3YwKK9Mr+96HYpEj/Of6zVI="
},
"host": {
"id": "4751091017865185079",
Expand Down
5 changes: 5 additions & 0 deletions packages/gcp/data_stream/compute/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
---
description: Pipeline for parsing GCP Compute metrics.
processors:
- fingerprint:
fields:
- gcp.labels
target_field: gcp.labels_fingerprint
ignore_missing: true
- rename:
field: gcp.metrics.firewall.dropped.bytes
target_field: gcp.compute.firewall.dropped.bytes
Expand Down
1 change: 1 addition & 0 deletions packages/gcp/data_stream/compute/fields/agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
type: keyword
ignore_above: 1024
description: Instance name of the host machine.
dimension: true
- name: project.id
type: keyword
description: Name of the project in Google Cloud.
Expand Down
9 changes: 9 additions & 0 deletions packages/gcp/data_stream/compute/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,24 @@
name: cloud
- external: ecs
name: cloud.account.id
dimension: true
- external: ecs
name: cloud.account.name
dimension: true
- external: ecs
name: cloud.availability_zone
dimension: true
- external: ecs
name: cloud.instance.id
dimension: true
- external: ecs
name: cloud.machine.type
dimension: true
- external: ecs
name: cloud.provider
- external: ecs
name: cloud.region
dimension: true
- external: ecs
name: ecs.version
- external: ecs
Expand All @@ -22,3 +28,6 @@
name: error.message
- external: ecs
name: service.type
- external: ecs
name: agent.id
dimension: true
6 changes: 6 additions & 0 deletions packages/gcp/data_stream/compute/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
- name: gcp
type: group
description: >-
GCP module
fields:
- name: labels_fingerprint
dimension: true
type: keyword
description: >-
Hashed value of the labels field.
- name: labels
type: group
description: >-
Expand Down
5 changes: 5 additions & 0 deletions packages/gcp/data_stream/dataproc/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
---
description: Pipeline for parsing GCP Dataproc metrics.
processors:
- fingerprint:
fields:
- gcp.labels
target_field: gcp.labels_fingerprint
ignore_missing: true
- rename:
field: gcp.metrics.batch.spark.executors.count
target_field: gcp.dataproc.batch.spark.executors.count
Expand Down
5 changes: 5 additions & 0 deletions packages/gcp/data_stream/dataproc/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@
name: cloud
- external: ecs
name: cloud.account.id
dimension: true
- external: ecs
name: cloud.account.name
dimension: true
- external: ecs
name: cloud.availability_zone
- external: ecs
Expand All @@ -22,3 +24,6 @@
name: error.message
- external: ecs
name: service.type
- external: ecs
name: agent.id
dimension: true
6 changes: 6 additions & 0 deletions packages/gcp/data_stream/dataproc/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
- name: gcp
type: group
description: >-
GCP module
fields:
- name: labels_fingerprint
dimension: true
type: keyword
description: >-
Hashed value of the labels field.
- name: labels
type: group
description: >-
Expand Down
5 changes: 5 additions & 0 deletions packages/gcp/data_stream/firestore/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
---
description: Pipeline for parsing GCP Firestore metrics.
processors:
- fingerprint:
fields:
- gcp.labels
target_field: gcp.labels_fingerprint
ignore_missing: true
- rename:
field: gcp.metrics.document.delete.count
target_field: gcp.firestore.document.delete.count
Expand Down
5 changes: 5 additions & 0 deletions packages/gcp/data_stream/firestore/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@
name: cloud
- external: ecs
name: cloud.account.id
dimension: true
- external: ecs
name: cloud.account.name
dimension: true
- external: ecs
name: cloud.availability_zone
- external: ecs
Expand All @@ -22,3 +24,6 @@
name: error.message
- external: ecs
name: service.type
- external: ecs
name: agent.id
dimension: true
6 changes: 6 additions & 0 deletions packages/gcp/data_stream/firestore/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
- name: gcp
type: group
description: >-
GCP module
fields:
- name: labels_fingerprint
dimension: true
type: keyword
description: >-
Hashed value of the labels field.
- name: labels
type: group
description: >-
Expand Down
5 changes: 5 additions & 0 deletions packages/gcp/data_stream/gke/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
---
description: Pipeline for parsing GCP GKE metrics.
processors:
- fingerprint:
fields:
- gcp.labels
target_field: gcp.labels_fingerprint
ignore_missing: true
- rename:
field: gcp.metrics.container.cpu.core_usage_time.sec
target_field: gcp.gke.container.cpu.core_usage_time.sec
Expand Down
5 changes: 5 additions & 0 deletions packages/gcp/data_stream/gke/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@
name: cloud
- external: ecs
name: cloud.account.id
dimension: true
- external: ecs
name: cloud.account.name
dimension: true
- external: ecs
name: cloud.availability_zone
- external: ecs
Expand All @@ -22,3 +24,6 @@
name: error.message
- external: ecs
name: service.type
- external: ecs
name: agent.id
dimension: true
6 changes: 6 additions & 0 deletions packages/gcp/data_stream/gke/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
- name: gcp
type: group
description: >-
GCP module
fields:
- name: labels_fingerprint
dimension: true
type: keyword
description: >-
Hashed value of the labels field.
- name: labels
type: group
description: >-
Expand Down
Loading