Support application default credentials (ADC) for Google Pub/Sub

[mwasilew2]

What does this PR do?

Update the Google Pub/Sub input to support reading Application Default Credentials
(ADC) in addition to the credentials_file and credentials_json config options.

If neither config option is set then it will attempt to search for the default credentials.
Generally this means reading the GOOGLE_APPLICATION_CREDENTIALS environment
variable plus searching a few other well known locations.

The googlecloud module was updates to support all three authentication mechanisms.

Why is it important?

Without this change, Filebeat requires a path to a file with credentials to be provided in its config and is unable to use for example a default service account.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

1. Upload the Filebeat binary to a GCP instance created with default config
2. Run the binary configured to read from PubSub, but do not provide any credentials
3. Filebeat should be able to detect that it's on GCP and use ADC (get a token for the default service account for the VM from the metadata server)

Related issues

• Closes Filebeat doesn't use ADC on GCP instances #15667

Use cases

Sending logs to Elastic from PubSub queue

Screenshots

Logs

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[blakerouse]

@elastic/siem Can you take a look at this change?

[andrewkroh]

jenkins, test this please

[andrewkroh]

The integration tests is failing due to this change. https://travis-ci.org/elastic/beats/jobs/638704764#L9301-L9327

[andrewkroh]

I think it would be good to enable the use of ADC. But I'd prefer to still have up-front validation that the ADC is available this way it can throw an error when filebeat check config runs.

So the validation would check that one of the options is available - ADC, credential_file exists, or JSON credentials is non-empty. Google has some sample code showing how to check for ADC at https://cloud.google.com/docs/authentication/production#obtaining_credentials_on_compute_engine_kubernetes_engine_app_engine_flexible_environment_and_cloud_functions. Perhaps you can try using the google.FindDefaultCredentials from that example to implement this validation.

[mwasilew2]

@andrewkroh

So the validation would check that one of the options is available - ADC, credential_file exists, or JSON credentials is non-empty.

yea, it definitely makes sense

I took another stab at it, let me know if this approach is better

I also tried running the integration tests locally but they seem to fail on netflow:

$ make -C x-pack/filebeat testsuite
make: Entering directory '/home/michal/go/src/github.com/elastic/beats/x-pack/filebeat'
Installing mage v1.9.0 from vendor dir.
go install -mod=vendor -ldflags="-X github.com/magefile/mage/mage.gitTag=v1.9.0" github.com/magefile/mage
/home/michal/.magefile cleaned
rm -f build/TEST-go-integration.out
mage update build unitTest integTest || ( cat build/TEST-go-integration.out && false )
No fields files for module azureeventhub
No fields files for module cloudfoundry
No fields files for module googlepubsub
No fields files for module httpjson
No fields files for module o365audit
Generated fields.yml for filebeat to /home/michal/go/src/github.com/elastic/beats/x-pack/filebeat/fields.yml
>> Building filebeat.yml for linux/amd64
>> Building filebeat.reference.yml for linux/amd64
>> Building filebeat.docker.yml for linux/amd64
exec: go list -m
>> build: Building filebeat
>> go test: Unit Testing
FAILURES:
Package: github.com/elastic/beats/v7/x-pack/filebeat/input/netflow
Test:    [build failed]
----
SUMMARY:
  Fail:     1
  Skip:     7
  Pass:     453
  Packages: 21
  Duration: 26.73016531s
  JUnit Report: /home/michal/go/src/github.com/elastic/beats/x-pack/filebeat/build/TEST-go-unit.xml
  Output File:  /home/michal/go/src/github.com/elastic/beats/x-pack/filebeat/build/TEST-go-unit.out
>> go test: Unit Test Failed
Error: go test failed: 1 test failures
cat: build/TEST-go-integration.out: No such file or directory
make: *** [../../dev-tools/make/xpack.mk:48: testsuite] Error 1
make: Leaving directory '/home/michal/go/src/github.com/elastic/beats/x-pack/filebeat'
[1]    59958 exit 2     make -C x-pack/filebeat testsuite

let me know if there's anything else I can help with

[andrewkroh]

I update the PR to add:

• unit test for the config validation
• updated google-pubsub doc to mention ADC can be used
• update googlecloud modules to support ADC and credentials_json

[andrewkroh]

run tests

[andrewkroh]

LGTM