Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync fork #13

Merged
merged 62 commits into from
Feb 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
1548707
add missing iam properties to factory_subnets
jamesdalf Feb 13, 2023
ebc4bc5
Workaround to mitigate issue 9164
lcaggio Feb 13, 2023
603bb56
Merge pull request #1154 from GoogleCloudPlatform/lcaggio/workaround-…
lcaggio Feb 14, 2023
7bbeac8
Add 'max_time_travel_hours ' on BQ module
lcaggio Feb 14, 2023
13f2859
Merge pull request #1156 from GoogleCloudPlatform/lcaggio/bq-max_time…
lcaggio Feb 14, 2023
742b5ba
Fix tfvars sample for fast bootstrap stage
juliocc Feb 14, 2023
e833485
Update main.tf (#1158)
chemapolo Feb 15, 2023
36a7347
FAST stage docs cleanup (#1145)
ludoo Feb 15, 2023
e64e8db
Allow additive IAM grants by robots name
wiktorn Feb 11, 2023
a853dc4
update Data Platform blueprint README with more example Dataflow comm…
aymanfarhat Feb 16, 2023
fcdadf5
Fix spelling mistake
aymanfarhat Feb 16, 2023
eac2065
Update on docs wording
aymanfarhat Feb 16, 2023
7c9841d
Merge pull request #1161 from aymanfarhat/enhancement/dpf-df-doc
lcaggio Feb 16, 2023
a497aef
feat: new version of projects-data-source based on AssetInventory ds
averbuks Feb 18, 2023
d970db0
Merge branch 'master' into projects-ds-new-version
averbuks Feb 18, 2023
1c302c7
TF formatting
averbuks Feb 18, 2023
8174890
Sort variables
averbuks Feb 18, 2023
ff4b2ff
Regenerate docs
averbuks Feb 18, 2023
befc73e
feat: Update TFC+WIF blueprint with TFC Dynamic Credentials feature.
averbuks Feb 18, 2023
5250877
Merge branch 'master' into projects-ds-new-version
averbuks Feb 19, 2023
4ad30b8
Fix tests for tf-cloud-dynamic-credentials blueprint
averbuks Feb 19, 2023
07957ce
Merge branch 'master' into projects-ds-new-version
averbuks Feb 19, 2023
f16511b
Fix typo in readme
averbuks Feb 19, 2023
0d0a2b4
Merge pull request #1163 from GoogleCloudPlatform/projects-ds-new-ver…
averbuks Feb 19, 2023
5cf60cb
Fix Terraform formatting and add module_prefix attribute to modules_c…
antonkovach Feb 19, 2023
5905903
fix module_prefix (#1164)
antonkovach Feb 19, 2023
ad6667a
First commit
lcaggio Feb 19, 2023
c8f2551
Fix variabler to support existing projects.
lcaggio Feb 19, 2023
db6a4f9
Remove variables
lcaggio Feb 19, 2023
f4c1fa6
Fix tests.
lcaggio Feb 19, 2023
c523dce
Fix
lcaggio Feb 20, 2023
eda9597
Fix linting
lcaggio Feb 20, 2023
63a81a9
Fix Fast test
lcaggio Feb 20, 2023
970b8ff
Fix DP Fast variables.
lcaggio Feb 20, 2023
2564c9b
Fix README
lcaggio Feb 20, 2023
840abd6
Merge branch 'master' into master
ludoo Feb 20, 2023
3085922
Fix tests
juliocc Feb 20, 2023
fed928b
Merge pull request #1153 from jamesdalf/master
juliocc Feb 20, 2023
0ca0b2e
Fix variable name
juliodiez Feb 20, 2023
bd91e97
Merge pull request #1166 from GoogleCloudPlatform/vpc-sc-fix
juliodiez Feb 20, 2023
4a9ce1c
Update README
juliodiez Feb 21, 2023
77df3c8
Merge pull request #1168 from GoogleCloudPlatform/juliodiez-patch-1
juliodiez Feb 21, 2023
6b767c9
Simplify org policies data model in resman modules.
juliocc Feb 21, 2023
d3bcf62
Update yaml org policies
juliocc Feb 21, 2023
62834ca
Update READMEs
juliocc Feb 21, 2023
a5e905c
Update remaining org policies
juliocc Feb 21, 2023
1696f70
Update PF variables for org policies
juliocc Feb 21, 2023
097e905
Merge pull request #1167 from GoogleCloudPlatform/jccb/org-policies-r…
juliocc Feb 21, 2023
2108b46
Fix Tests, rely on iam additive.
lcaggio Feb 22, 2023
ac75cbe
Fix lint.
lcaggio Feb 22, 2023
ad08406
Add documentation about referring modules stored on CSR
wiktorn Feb 22, 2023
e39be7b
Fix
lcaggio Feb 22, 2023
b279c08
Fix README and IAM files
lcaggio Feb 23, 2023
47855cb
Merge branch 'master' into lcaggio/dp-projectcreate
lcaggio Feb 23, 2023
89da766
Merge pull request #1165 from GoogleCloudPlatform/lcaggio/dp-projectc…
lcaggio Feb 23, 2023
08ba94a
Allow to not use any health check
juliodiez Feb 24, 2023
ae6080e
Modifications related to autopilot and workload identity. Added workl…
apichick Feb 24, 2023
dd416a4
Merge pull request #1171 from apichick/gke-autopilot-workload-identity
apichick Feb 24, 2023
5a44baa
Merge branch 'master' into juliodiez-patch-1
juliodiez Feb 24, 2023
afb2970
Merge pull request #1172 from GoogleCloudPlatform/juliodiez-patch-1
juliodiez Feb 24, 2023
3b02234
Don't define nor use health checks with SNEGs
juliodiez Feb 24, 2023
72a1189
Merge pull request #1174 from GoogleCloudPlatform/juliodiez-patch-1
juliodiez Feb 24, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion blueprints/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ This section provides **[networking blueprints](./networking/)** that implement
Currently available blueprints:

- **apigee** - [Apigee Hybrid on GKE](./apigee/hybrid-gke/), [Apigee X analytics in BigQuery](./apigee/bigquery-analytics), [Apigee network patterns](./apigee/network-patterns/)
- **cloud operations** - [Active Directory Federation Services](./cloud-operations/adfs), [Cloud Asset Inventory feeds for resource change tracking and remediation](./cloud-operations/asset-inventory-feed-remediation), [Fine-grained Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Cloud DNS & Shared VPC design](./cloud-operations/dns-shared-vpc), [Delegated Role Grants](./cloud-operations/iam-delegated-role-grants), [Networking Dashboard](./cloud-operations/network-dashboard), [Managing on-prem service account keys by uploading public keys](./cloud-operations/onprem-sa-key-management), [Compute Image builder with Hashicorp Packer](./cloud-operations/packer-image-builder), [Packer example](./cloud-operations/packer-image-builder/packer), [Compute Engine quota monitoring](./cloud-operations/quota-monitoring), [Scheduled Cloud Asset Inventory Export to Bigquery](./cloud-operations/scheduled-asset-inventory-export-bq), [Configuring workload identity federation for Terraform Cloud/Enterprise workflow](./cloud-operations/terraform-enterprise-wif), [TCP healthcheck and restart for unmanaged GCE instances](./cloud-operations/unmanaged-instances-healthcheck), [Migrate for Compute Engine (v5) blueprints](./cloud-operations/vm-migration), [Configuring workload identity federation to access Google Cloud resources from apps running on Azure](./cloud-operations/workload-identity-federation)
- **cloud operations** - [Active Directory Federation Services](./cloud-operations/adfs), [Cloud Asset Inventory feeds for resource change tracking and remediation](./cloud-operations/asset-inventory-feed-remediation), [Fine-grained Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Cloud DNS & Shared VPC design](./cloud-operations/dns-shared-vpc), [Delegated Role Grants](./cloud-operations/iam-delegated-role-grants), [Networking Dashboard](./cloud-operations/network-dashboard), [Managing on-prem service account keys by uploading public keys](./cloud-operations/onprem-sa-key-management), [Compute Image builder with Hashicorp Packer](./cloud-operations/packer-image-builder), [Packer example](./cloud-operations/packer-image-builder/packer), [Compute Engine quota monitoring](./cloud-operations/quota-monitoring), [Scheduled Cloud Asset Inventory Export to Bigquery](./cloud-operations/scheduled-asset-inventory-export-bq), [Configuring workload identity federation with Terraform Cloud/Enterprise workflows](./cloud-operations/terraform-cloud-dynamic-credentials), [TCP healthcheck and restart for unmanaged GCE instances](./cloud-operations/unmanaged-instances-healthcheck), [Migrate for Compute Engine (v5) blueprints](./cloud-operations/vm-migration), [Configuring workload identity federation to access Google Cloud resources from apps running on Azure](./cloud-operations/workload-identity-federation)
- **data solutions** - [GCE and GCS CMEK via centralized Cloud KMS](./data-solutions/cmek-via-centralized-kms), [Cloud Composer version 2 private instance, supporting Shared VPC and external CMEK key](./data-solutions/composer-2), [Cloud SQL instance with multi-region read replicas](./data-solutions/cloudsql-multiregion), [Data Platform](./data-solutions/data-platform-foundations), [Spinning up a foundation data pipeline on Google Cloud using Cloud Storage, Dataflow and BigQuery](./data-solutions/gcs-to-bq-with-least-privileges), [#SQL Server Always On Groups blueprint](./data-solutions/sqlserver-alwayson), [Data Playground](./data-solutions/data-playground), [MLOps with Vertex AI](./data-solutions/vertex-mlops), [Shielded Folder](./data-solutions/shielded-folder)
- **factories** - [The why and the how of Resource Factories](./factories), [Google Cloud Identity Group Factory](./factories/cloud-identity-group-factory), [Google Cloud BQ Factory](./factories/bigquery-factory), [Google Cloud VPC Firewall Factory](./factories/net-vpc-firewall-yaml), [Minimal Project Factory](./factories/project-factory)
- **GKE** - [Binary Authorization Pipeline Blueprint](./gke/binauthz), [Storage API](./gke/binauthz/image), [Multi-cluster mesh on GKE (fleet API)](./gke/multi-cluster-mesh-gke-fleet-api), [GKE Multitenant Blueprint](./gke/multitenant-fleet), [Shared VPC with GKE support](./networking/shared-vpc-gke/)
Expand Down
4 changes: 2 additions & 2 deletions blueprints/cloud-operations/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,9 @@ This [blueprint](./onprem-sa-key-management) shows how to manage IAM Service Acc

<br clear="left">

## Workload identity federation for Terraform Enterprise workflow
## Workload identity federation with Terraform Cloud workflows

<a href="./terraform-enterprise-wif" title="Workload identity federation for Terraform Cloud/Enterprise workflow"><img src="./terraform-enterprise-wif/diagram.png" align="left" width="280px"></a> This [blueprint](./terraform-enterprise-wif) shows how to configure [Wokload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) between [Terraform Cloud/Enterprise](https://developer.hashicorp.com/terraform/enterprise) instance and Google Cloud.
<a href="./terraform-cloud-dynamic-credentials" title="Workload identity federation with Terraform Cloud/Enterprise workflows"><img src="./terraform-cloud-dynamic-credentials/diagram.png" align="left" width="280px"></a> This [blueprint](./terraform-cloud-dynamic-credentials) shows how to configure [Wokload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) between [Terraform Cloud/Enterprise](https://developer.hashicorp.com/terraform/enterprise) instance and Google Cloud.

<br clear="left">

Expand Down
Original file line number Diff line number Diff line change
@@ -1,27 +1,27 @@
# Configuring workload identity federation for Terraform Cloud/Enterprise workflow
# Configuration of workload identity federation for Terraform Cloud/Enterprise workflows

The most common way to use Terraform Cloud for GCP deployments is to store a GCP Service Account Key as a part of TFE Workflow configuration, as we all know there are security risks due to the fact that keys are long term credentials that could be compromised.
The most common way to use Terraform Cloud for GCP deployments is to store a GCP Service Account Key as a part of TFC Workflow configuration, as we all know there are security risks due to the fact that keys are long term credentials that could be compromised.

Workload identity federation enables applications running outside of Google Cloud to replace long-lived service account keys with short-lived access tokens. This is achieved by configuring Google Cloud to trust an external identity provider, so applications can use the credentials issued by the external identity provider to impersonate a service account.

This blueprint shows how to set up [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) between [Terraform Cloud/Enterprise](https://developer.hashicorp.com/terraform/enterprise) instance and Google Cloud. This will be possible by configuring workload identity federation to trust oidc tokens generated for a specific workflow in a Terraform Enterprise organization.
This blueprint shows how to set up [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) between [Terraform Cloud/Enterprise](https://developer.hashicorp.com/terraform/enterprise) instance and Google Cloud. This will be possible by configuring workload identity federation and [Terraform Cloud Dynamic Provider Credentials](https://www.hashicorp.com/blog/terraform-cloud-adds-dynamic-provider-credentials-vault-official-cloud-providers).

The following diagram illustrates how the VM will get a short-lived access token and use it to access a resource:

![Sequence diagram](diagram.png)

## Running the blueprint

### Create Terraform Enterprise Workflow
If you don't have an existing Terraform Enterprise organization you can sign up for a [free trial](https://app.terraform.io/public/signup/account) account.
### Create Terraform Cloud Workflow
If you don't have an existing Terraform Cloud organization you can sign up for a [free trial](https://app.terraform.io/public/signup/account) account.

Create a new Workspace for a `CLI-driven workflow` (Identity Federation will work for any workflow type, but for simplicity of the blueprint we use CLI driven workflow).

Note workspace name and id (id starts with `ws-`), we will use them on a later stage.

Go to the organization settings and note the org name and id (id starts with `org-`).

### Deploy GCP Workload Identity Pool Provider for Terraform Enterprise
### Deploy GCP Workload Identity Pool Provider for Terraform Cloud integration

> **_NOTE:_** This is a preparation part and should be executed on behalf of a user with enough permissions.

Expand All @@ -32,7 +32,7 @@ Required permissions when new project is created:
- Workload Identity Admin on the project level
- Project IAM Admin on the project level

Fill out required variables, use TFE Org and Workspace IDs from the previous steps (IDs are not the names).
Fill out required variables, use TFC Org and Workspace IDs from the previous steps (IDs are not the names).
```bash
cd gcp-workload-identity-provider

Expand All @@ -50,34 +50,41 @@ terraform init
terraform apply
```

As a result a set of outputs will be provided (your values will be different), note the output since we will use it on the next steps.
You will receive a set of outputs (your values may be different), note them because we will need them in the next steps.

```
impersonate_service_account_email = "[email protected]"
project_id = "tfe-test-oidc"
workload_identity_audience = "//iam.googleapis.com/projects/476538149566/locations/global/workloadIdentityPools/tfe-pool/providers/tfe-provider"
workload_identity_pool_provider_id = "projects/476538149566/locations/global/workloadIdentityPools/tfe-pool/providers/tfe-provider"
project_id = "tfc-dynamic-creds-gcp"
tfc_workspace_wariables = {
"TFC_GCP_PROJECT_NUMBER" = "200635100209"
"TFC_GCP_PROVIDER_AUTH" = "true"
"TFC_GCP_RUN_SERVICE_ACCOUNT_EMAIL" = "[email protected]"
"TFC_GCP_WORKLOAD_POOL_ID" = "tfc-pool"
"TFC_GCP_WORKLOAD_PROVIDER_ID" = "tfc-provider"
}
```

### Configure OIDC provider for your TFE Workflow
### Configure Dynamic Provider Credentials for your TFC Workflow

To enable OIDC for a TFE workflow it's enough to setup an environment variable `TFC_WORKLOAD_IDENTITY_AUDIENCE`.
To configure [GCP Dynamic Provider Credentials](https://developer.hashicorp.com/terraform/cloud-docs/workspaces/dynamic-provider-credentials/gcp-configuration) for a TFC workflow, you need to set a set of environment variables:
- `TFC_GCP_PROVIDER_AUTH`
- `TFC_GCP_PROJECT_NUMBER`
- `TFC_GCP_RUN_SERVICE_ACCOUNT_EMAIL`
- `TFC_GCP_WORKLOAD_POOL_ID`
- `TFC_GCP_WORKLOAD_PROVIDER_ID`

Go the the Workflow -> Variables and add a new variable `TFC_WORKLOAD_IDENTITY_AUDIENCE` equal to the value of `workload_identity_audience` output, in our example it's:
Go to the Workflow -> Variables page and click the + Add variable button. For variable type select ` Environment variable`. The variable names listed above are the names of the variables that you need to set. The values provided in the terraform output in the previous step are the values that you need to provide for each variable.

```
TFC_WORKLOAD_IDENTITY_AUDIENCE = "//iam.googleapis.com/projects/476538149566/locations/global/workloadIdentityPools/tfe-pool/providers/tfe-provider"
```

At that point we setup GCP Identity Federation to trust TFE generated OIDC tokens, so the TFE workflow can use the token to impersonate a GCP Service Account.
At that point we set up GCP Identity Federation to trust TFC generated OIDC tokens, workflow should be able to use Dynamic Provider Credentials to impersonate a GCP Service Account.

## Testing the blueprint

In order to test the setup we will deploy a GCS bucket from TFE Workflow using OIDC token for Service Account Impersonation.
To test the setup, we will deploy a GCS bucket from the TFC Workflow created in the previous step.

This will allow us to verify that the workflow can successfully interact with GCP services using the TFC Dynamic Provider Credentials.

### Configure backend and variables

First, we need to configure TFE Remote backend for our testing terraform code, use TFE Organization name and workspace name (names are not the same as ids)
First, we need to configure the TFC Remote backend for our testing Terraform code. Use the TFC Organization name and workspace name (names are not the same as ids).

```
cd ../tfc-workflow-using-wif
Expand All @@ -89,7 +96,7 @@ vi backend.tf

```

Fill out variables based on the output from the preparation steps:
Fill out `project_id` variable based on the output from the preparation steps:

```
mv terraform.auto.tfvars.template terraform.auto.tfvars
Expand All @@ -100,7 +107,7 @@ vi terraform.auto.tfvars

### Authenticate terraform for triggering CLI-driven workflow

Follow this [documentation](https://learn.hashicorp.com/tutorials/terraform/cloud-login) to login ti terraform cloud from the CLI.
Follow this [documentation](https://learn.hashicorp.com/tutorials/terraform/cloud-login) to login to terraform cloud from the CLI.

### Trigger the workflow

Expand All @@ -110,6 +117,6 @@ terraform init
terraform apply
```

As a result we have a successfully deployed GCS bucket from Terraform Enterprise workflow using Workload Identity Federation.
As a result we have a successfully deployed GCS bucket from Terraform Cloud workflow using Workload Identity Federation.

Once done testing, you can clean up resources by running `terraform destroy` first in the `tfc-workflow-using-wif` and then `gcp-workload-identity-provider` folders.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -1,10 +1,14 @@
# GCP Workload Identity Provider for Terraform Enterprise
# GCP Workload Identity Provider for Terraform Cloud Dynamic Credentials

This terraform code is a part of [GCP Workload Identity Federation for Terraform Enterprise](../) blueprint.
This terraform code is a part of [GCP Workload Identity Federation for Terraform Cloud](../) blueprint.

The codebase provisions the following list of resources:

- GCS Bucket
- (optional) GCP Project
- IAM Service Account
- Workload Identity Pool
- Workload Identity Provider
- IAM Permissins
<!-- BEGIN TFDOC -->

## Variables
Expand All @@ -13,21 +17,19 @@ The codebase provisions the following list of resources:
|---|---|:---:|:---:|:---:|
| [billing_account](variables.tf#L16) | Billing account id used as default for new projects. | <code>string</code> | ✓ | |
| [project_id](variables.tf#L43) | Existing project id. | <code>string</code> | ✓ | |
| [tfe_organization_id](variables.tf#L48) | TFE organization id. | <code>string</code> | ✓ | |
| [tfe_workspace_id](variables.tf#L53) | TFE workspace id. | <code>string</code> | ✓ | |
| [issuer_uri](variables.tf#L21) | Terraform Enterprise uri. Replace the uri if a self hosted instance is used. | <code>string</code> | | <code>&#34;https:&#47;&#47;app.terraform.io&#47;&#34;</code> |
| [tfc_organization_id](variables.tf#L48) | TFC organization id. | <code>string</code> | ✓ | |
| [tfc_workspace_id](variables.tf#L53) | TFC workspace id. | <code>string</code> | ✓ | |
| [issuer_uri](variables.tf#L21) | Terraform Cloud/Enterprise uri. Replace the uri if a self hosted instance is used. | <code>string</code> | | <code>&#34;https:&#47;&#47;app.terraform.io&#47;&#34;</code> |
| [parent](variables.tf#L27) | Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format. | <code>string</code> | | <code>null</code> |
| [project_create](variables.tf#L37) | Create project instead of using an existing one. | <code>bool</code> | | <code>true</code> |
| [workload_identity_pool_id](variables.tf#L58) | Workload identity pool id. | <code>string</code> | | <code>&#34;tfe-pool&#34;</code> |
| [workload_identity_pool_provider_id](variables.tf#L64) | Workload identity pool provider id. | <code>string</code> | | <code>&#34;tfe-provider&#34;</code> |
| [workload_identity_pool_id](variables.tf#L58) | Workload identity pool id. | <code>string</code> | | <code>&#34;tfc-pool&#34;</code> |
| [workload_identity_pool_provider_id](variables.tf#L64) | Workload identity pool provider id. | <code>string</code> | | <code>&#34;tfc-provider&#34;</code> |

## Outputs

| name | description | sensitive |
|---|---|:---:|
| [impersonate_service_account_email](outputs.tf#L16) | Service account to be impersonated by workload identity. | |
| [project_id](outputs.tf#L21) | GCP Project ID. | |
| [workload_identity_audience](outputs.tf#L26) | TFC Workload Identity Audience. | |
| [workload_identity_pool_provider_id](outputs.tf#L31) | GCP workload identity pool provider ID. | |
| [project_id](outputs.tf#L15) | GCP Project ID. | |
| [tfc_workspace_wariables](outputs.tf#L20) | Variables to be set on the TFC workspace. | |

<!-- END TFDOC -->
Loading