[Metricbeat] Add billing metricset into googlecloud module

CHANGELOG.next.asciidoc

@@ -662,6 +662,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add host inventory metrics to azure compute_vm metricset. {pull}20641[20641]
 - Add host inventory metrics to googlecloud compute metricset. {pull}20391[20391]
 - Request prometheus endpoints to be gzipped by default {pull}20766[20766]
+- Add billing metricset into googlecloud module. {pull}20812[20812] {issue}20738[20738]
 
 *Packetbeat*
 

go.mod

@@ -5,6 +5,7 @@ go 1.14
 require (
 	4d63.com/tz v1.1.1-0.20191124060701-6d37baae851b
 	cloud.google.com/go v0.51.0
+	cloud.google.com/go/bigquery v1.0.1
 	cloud.google.com/go/pubsub v1.0.1
 	cloud.google.com/go/storage v1.0.0
 	code.cloudfoundry.org/go-diodes v0.0.0-20190809170250-f77fb823c7ee // indirect

metricbeat/docs/fields.asciidoc

@@ -18426,6 +18426,48 @@ type: object
 
 --
 
+[float]
+=== billing
+
+Google Cloud Billing metrics
+
+
+*`googlecloud.billing.cost_type`*::
++
+--
+Cost types include regular, tax, adjustment, and rounding_error.
+
+type: keyword
+
+--
+
+*`googlecloud.billing.invoice_month`*::
++
+--
+Billing report month.
+
+type: keyword
+
+--
+
+*`googlecloud.billing.project_id`*::
++
+--
+Project ID of the billing report belongs to.
+
+type: keyword
+
+--
+
+*`googlecloud.billing.total`*::
++
+--
+Total billing amount.
+
+type: float
+
+--
+
 [float]
 === compute
 

metricbeat/docs/modules/googlecloud.asciidoc

@@ -152,6 +152,17 @@ For example, if Compute Metricset fetches 14 metrics (which is the number of met
 Currently, we have `compute`, `loadbalancing`, `pubsub`, `metrics` and
 `storage` metricset in `googlecloud` module.
 
+[float]
+=== `billing`
+This metricset fetches billing metrics from https://cloud.google.com/bigquery[GCP BigQuery]
+Cloud Billing allows users to export billing data into BigQuery automatically
+throughout the day. This metricset gets access to the daily cost detail table
+periodically to export billing metrics for further analysis.
+
+The `billing` metricset comes with a predefined dashboard:
+
+image::./images/metricbeat-googlecloud-billing-overview.png[]
+
 [float]
 === `compute`
 This metricset fetches metrics from https://cloud.google.com/compute/[Compute Engine]
@@ -274,13 +285,25 @@ metricbeat.modules:
         - "instance/cpu/usage_time"
         - "instance/cpu/utilization"
         - "instance/uptime"
+
+- module: googlecloud
+  metricsets:
+    - billing
+  period: 24h
+  project_id: "your project id"
+  credentials_file_path: "your JSON credentials file path"
+  dataset_id: "dataset id"
+  table_pattern: "table pattern"
+  cost_type: "regular"
 ----
 
 [float]
 === Metricsets
 
 The following metricsets are available:
 
+* <<metricbeat-metricset-googlecloud-billing,billing>>
+
 * <<metricbeat-metricset-googlecloud-compute,compute>>
 
 * <<metricbeat-metricset-googlecloud-loadbalancing,loadbalancing>>
@@ -291,6 +314,8 @@ The following metricsets are available:
 
 * <<metricbeat-metricset-googlecloud-storage,storage>>
 
+include::googlecloud/billing.asciidoc[]
+
 include::googlecloud/compute.asciidoc[]
 
 include::googlecloud/loadbalancing.asciidoc[]

metricbeat/docs/modules/googlecloud/billing.asciidoc

@@ -0,0 +1,24 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+[[metricbeat-metricset-googlecloud-billing]]
+[role="xpack"]
+=== Google Cloud Platform billing metricset
+
+beta[]
+
+include::../../../../x-pack/metricbeat/module/googlecloud/billing/_meta/docs.asciidoc[]
+
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-googlecloud,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../../x-pack/metricbeat/module/googlecloud/billing/_meta/data.json[]
+----

metricbeat/docs/modules_list.asciidoc

@@ -111,7 +111,8 @@ This file is generated! See scripts/mage/docs_collector.go
 .2+| .2+|  |<<metricbeat-metricset-golang-expvar,expvar>>   
 |<<metricbeat-metricset-golang-heap,heap>>   
 |<<metricbeat-module-googlecloud,Google Cloud Platform>>  beta[]   |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
-.5+| .5+|  |<<metricbeat-metricset-googlecloud-compute,compute>> beta[]  
+.6+| .6+|  |<<metricbeat-metricset-googlecloud-billing,billing>> beta[]  
+|<<metricbeat-metricset-googlecloud-compute,compute>> beta[]  
 |<<metricbeat-metricset-googlecloud-loadbalancing,loadbalancing>> beta[]  
 |<<metricbeat-metricset-googlecloud-metrics,metrics>> beta[]  
 |<<metricbeat-metricset-googlecloud-pubsub,pubsub>> beta[]  

x-pack/metricbeat/metricbeat.reference.yml

@@ -555,6 +555,16 @@ metricbeat.modules:
         - "instance/cpu/utilization"
         - "instance/uptime"
 
+- module: googlecloud
+  metricsets:
+    - billing
+  period: 24h
+  project_id: "your project id"
+  credentials_file_path: "your JSON credentials file path"
+  dataset_id: "dataset id"
+  table_pattern: "table pattern"
+  cost_type: "regular"
+
 #------------------------------- Graphite Module -------------------------------
 - module: graphite
   metricsets: ["server"]

x-pack/metricbeat/module/googlecloud/_meta/config.yml

@@ -40,3 +40,13 @@
         - "instance/cpu/usage_time"
         - "instance/cpu/utilization"
         - "instance/uptime"
+
+- module: googlecloud
+  metricsets:
+    - billing
+  period: 24h
+  project_id: "your project id"
+  credentials_file_path: "your JSON credentials file path"
+  dataset_id: "dataset id"
+  table_pattern: "table pattern"
+  cost_type: "regular"

x-pack/metricbeat/module/googlecloud/_meta/docs.asciidoc

@@ -142,6 +142,17 @@ For example, if Compute Metricset fetches 14 metrics (which is the number of met
 Currently, we have `compute`, `loadbalancing`, `pubsub`, `metrics` and
 `storage` metricset in `googlecloud` module.
 
+[float]
+=== `billing`
+This metricset fetches billing metrics from https://cloud.google.com/bigquery[GCP BigQuery]
+Cloud Billing allows users to export billing data into BigQuery automatically
+throughout the day. This metricset gets access to the daily cost detail table
+periodically to export billing metrics for further analysis.
+
+The `billing` metricset comes with a predefined dashboard:
+
+image::./images/metricbeat-googlecloud-billing-overview.png[]
+
 [float]
 === `compute`
 This metricset fetches metrics from https://cloud.google.com/compute/[Compute Engine]