Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use GCP ADC for auth in Terraform Deployer #662

Closed
wants to merge 13 commits into from
+4,259 −12
Closed

Use GCP ADC for auth in Terraform Deployer #662

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
eba5f51
use GCP ADC for auth in Terraform Deployer
endorama Jan 24, 2022
a726836
add gcp test package
endorama Jan 27, 2022
d65cae2
remove missing data_stream docs
endorama Feb 1, 2022
1834d21
remove missing data_stream docs
endorama Feb 1, 2022
73d2317
fix linting
endorama Feb 1, 2022
cfb8674
leftover bash x options
endorama Feb 8, 2022
e1e3f4d
move GCP auth to function
endorama Feb 8, 2022
a14abb0
split main from functions
endorama Feb 8, 2022
2860bce
remove docker test deployment
endorama Feb 8, 2022
8bc1d37
add gcp.billing tests
endorama Feb 8, 2022
d908897
format file
endorama Feb 8, 2022
e670cce
remove unneded file
endorama Feb 8, 2022
cf19a6c
remove .gitignore
endorama Feb 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 28 additions & 12 deletions internal/install/_static/terraform_deployer_run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,41 @@

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any container logs for the GCP test package in beats-ci-temp-internal/Ingest-manager/elastic-package/PR-662-16/insecure-logs/gcp. Is it intended?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No is not, from the pipeline logs it seems it not running system tests (command start at line 244, hidden by the task header)

I set up the env.yml file in data_stream/billing/_dev/deploy/tf. Is this not enough to run system tests?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked your branch and you haven't configured any test policies. Test policies are used by elastic-package during system tests. Otherwise the elastic-package won't know what you re trying to test. It's covered in our manual.

Please take a look at the AWS test package and ec2_metrics tests.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test package with first test case (not requiring changes in this PR) added in #701

set -euo pipefail

# Terraform code may rely on content from other files than .tf files (es json, zip, html, text), so we copy all the content over
# See more: https://github.com/elastic/elastic-package/pull/603
cp -r /stage/* /workspace

cleanup() {
r=$?

set -x
terraform destroy -auto-approve

exit $r
}
trap cleanup EXIT INT TERM

terraform init
terraform plan
terraform apply -auto-approve && touch /tmp/tf-applied
gcp_auth() {
if test -n "$(printenv "GOOGLE_CREDENTIALS")"; then
# Save GCP credentials on disk and perform authentication
# NOTE: this is required for bq (and maybe other gcloud related tools) to authenticate
export "GOOGLE_APPLICATION_CREDENTIALS=/root/.config/gcloud/application_default_credentials.json"
printenv "GOOGLE_CREDENTIALS" > "$GOOGLE_APPLICATION_CREDENTIALS"
gcloud auth login --cred-file "$GOOGLE_APPLICATION_CREDENTIALS"
# NOTE: Terraform support authentication through GOOGLE_CREDENTIALS and usual gcloud ADC but other
# tools (like bq) don't support the first, so we always rely on gcloud ADC.
unset "GOOGLE_CREDENTIALS"
fi
}

if [[ "${BASH_SOURCE[0]}" = "$0" ]]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this condition required here?

Copy link
Member Author

@endorama endorama Feb 11, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not required, but it avoid running the code in case the file is sourced and identify a "main" like are, like the main function in go or C. From that point onward all code will be executed only when the file is explicitly executed (bash file.sh or ./file.sh). It helps grouping the relevant "main" code in the same area, preventing arbitrary code being added between functions.

Consider it a good way of writing BASH files, as it aids reading them (like __file__ == __main__ does for Python)

Copy link
Contributor

@mtojek mtojek Feb 11, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand your point of view, but please keep it simple as it was before. This file is intended to be called by container engine.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is run by container engine but is still read by developers :) The point was to simplifying reading the code flow. Anyway I removed it (will push once rebased onto main)

trap cleanup EXIT INT TERM

# Terraform code may rely on content from other files than .tf files (es json, zip, html, text), so we copy all the content over
# See more: https://github.com/elastic/elastic-package/pull/603
cp -r /stage/* /workspace

gcp_auth

terraform init
terraform plan
terraform apply -auto-approve && touch /tmp/tf-applied

echo "Terraform definitions applied."
echo "Terraform definitions applied."

set +x
mtojek marked this conversation as resolved.
Show resolved Hide resolved
while true; do sleep 1; done # wait for ctrl-c
while true; do sleep 1; done # wait for ctrl-c
fi
3 changes: 3 additions & 0 deletions test/packages/parallel/gcp/_dev/build/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
dependencies:
ecs:
reference: [email protected]
22 changes: 22 additions & 0 deletions test/packages/parallel/gcp/_dev/build/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Google Cloud Platform Integration

This integration is used to fetches logs and metrics from
[Google Cloud Platform](https://cloud.google.com/).

## GCP Credentials
GCP credentials are required for running GCP integration.

### Configuration parameters
* *project_id*: ID of the GCP project.
* *credentials_file*: Path to JSON file with GCP credentials. Required when not using `credentials_json`.
* *credentials_json*: Raw JSON text of GCP Credentials. Required when not using `credentials_file`.

#### Data stream specific configuration parameters
* *period*: How often the data stream is executed.
* *region*: Specify which GCP regions to query metrics from. If the `region`
is not set in the config, then by default, the integration will query metrics
from all available GCP regions. If both `region` and `zone` is set, `region` takes precedent.
* *zone*: Specify which GCP zones to query metrics from. If the `zone`
is not set in the config, then by default, the integration will query metrics
from all available GCP zone. If both `region` and `zone` is set, `region` takes precedent.
* *exclude_labels*: Exclude additional labels from metrics. Defaults to false.
9 changes: 9 additions & 0 deletions test/packages/parallel/gcp/_dev/build/docs/billing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Billing

## Metrics

This is the `billing` dataset.

{{event "billing"}}

{{fields "billing"}}
9 changes: 9 additions & 0 deletions test/packages/parallel/gcp/_dev/build/docs/compute.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Compute

## Metrics

This is the `compute` dataset.

{{event "compute"}}

{{fields "compute"}}
75 changes: 75 additions & 0 deletions test/packages/parallel/gcp/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# newer versions go on top
- version: "1.2.0"
changes:
- description: Add 8.0.0 version constraint
type: enhancement
link: https://github.com/elastic/integrations/pull/2251
- description: Add GCP Billing Metricset
type: enhancement
link: https://github.com/elastic/integrations/pull/2141
- description: Add GCP Compute Metricset
type: enhancement
link: https://github.com/elastic/integrations/pull/2301
- version: "1.1.2"
changes:
- description: Update Title and Description.
type: enhancement
link: https://github.com/elastic/integrations/pull/1965
- version: "1.1.1"
changes:
- description: Fix logic that checks for the 'forwarded' tag
type: bugfix
link: https://github.com/elastic/integrations/pull/1818
- version: "1.1.0"
changes:
- description: Update to ECS 1.12.0
type: enhancement
link: https://github.com/elastic/integrations/pull/1661
- version: "1.0.0"
changes:
- description: Move from experimental to GA
type: enhancement
link: https://github.com/elastic/integrations/pull/1568
- description: remove experimental from data_sets
type: enhancement
link: https://github.com/elastic/integrations/pull/1717
- version: "0.3.3"
changes:
- description: Convert to generated ECS fields
type: enhancement
link: https://github.com/elastic/integrations/pull/1478
- version: '0.3.2'
changes:
- description: update to ECS 1.11.0
type: enhancement
link: https://github.com/elastic/integrations/pull/1385
- version: "0.3.1"
changes:
- description: Escape special characters in docs
type: enhancement
link: https://github.com/elastic/integrations/pull/1405
- version: "0.3.0"
changes:
- description: Update integration description
type: enhancement
link: https://github.com/elastic/integrations/pull/1364
- version: "0.2.0"
changes:
- description: Set "event.module" and "event.dataset"
type: enhancement
link: https://github.com/elastic/integrations/pull/1240
- version: "0.1.0"
changes:
- description: update to ECS 1.10.0 and adding event.original options
type: enhancement
link: https://github.com/elastic/integrations/pull/1045
- version: "0.0.2"
changes:
- description: update to ECS 1.9.0
type: enhancement
link: https://github.com/elastic/integrations/pull/846
- version: "0.0.1"
changes:
- description: initial release
type: enhancement # can be one of: enhancement, bugfix, breaking-change
link: https://github.com/elastic/integrations/pull/459
38 changes: 38 additions & 0 deletions test/packages/parallel/gcp/data_stream/billing/_dev/deploy/tf/.terraform.lock.hcl
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading