Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IAM interface refactor #1595

Merged
merged 86 commits into from
Aug 20, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
bce3742
IAM modules refactor proposal
ludoo Aug 16, 2023
9cae7f1
policy
ludoo Aug 16, 2023
d1a7723
subheading
ludoo Aug 16, 2023
715a611
Update 20230816-iam-refactor.md
ludoo Aug 16, 2023
0b9b9d0
log Julio's +1
ludoo Aug 16, 2023
22aed28
data-catalog-policy-tag
ludoo Aug 16, 2023
82adf1f
dataproc
ludoo Aug 16, 2023
b416379
dataproc
ludoo Aug 16, 2023
faac3c0
folder
ludoo Aug 16, 2023
07c2532
folder
ludoo Aug 16, 2023
c597cbf
folder
ludoo Aug 16, 2023
455d33d
folder
ludoo Aug 16, 2023
9f5fbe4
project
ludoo Aug 16, 2023
64415f8
better filtering in test examples
ludoo Aug 16, 2023
2c12d3e
project
ludoo Aug 16, 2023
2bf0b92
folder
ludoo Aug 16, 2023
b919abb
folder
ludoo Aug 16, 2023
c584b99
organization
ludoo Aug 16, 2023
df4ae07
fix variable descriptions
ludoo Aug 16, 2023
bcf7bba
kms
ludoo Aug 16, 2023
832c8a3
net-vpc
ludoo Aug 16, 2023
31c1f33
dataplex-datascan
ludoo Aug 16, 2023
d31ce77
modules/iam-service-account
ludoo Aug 16, 2023
17575d3
modules/source-repository/
ludoo Aug 16, 2023
9df16af
blueprints/cloud-operations/vm-migration/
ludoo Aug 16, 2023
e9e6663
blueprints/third-party-solutions/wordpress
ludoo Aug 16, 2023
c555e14
dataplex-datascan
ludoo Aug 16, 2023
255bead
blueprints/cloud-operations/workload-identity-federation
ludoo Aug 16, 2023
f2762e0
blueprints/data-solutions/cloudsql-multiregion/
ludoo Aug 16, 2023
b86c245
blueprints/data-solutions/composer-2
ludoo Aug 16, 2023
b0ead3b
Update 20230816-iam-refactor.md
ludoo Aug 17, 2023
13996d1
Update 20230816-iam-refactor.md
ludoo Aug 17, 2023
2a2051e
Merge branch 'ludo/modules-iam-design' of github.com:GoogleCloudPlatf…
ludoo Aug 17, 2023
e89fede
capture discussion in architectural doc
ludoo Aug 17, 2023
9643017
update variable names and refactor proposal
ludoo Aug 17, 2023
1416020
project
ludoo Aug 17, 2023
3dac2c4
blueprints first round
ludoo Aug 17, 2023
adf50df
folder
ludoo Aug 17, 2023
64587af
organization
ludoo Aug 17, 2023
e498f38
data-catalog-policy-tag
ludoo Aug 17, 2023
0f194b6
re-enable folder inventory
ludoo Aug 17, 2023
fecd87a
project module style fix
ludoo Aug 17, 2023
17ad430
dataproc
ludoo Aug 17, 2023
5c7639b
source-repository
ludoo Aug 17, 2023
be5bef5
source-repository tests
ludoo Aug 17, 2023
9f10500
dataplex-datascan
ludoo Aug 17, 2023
d0ad27e
dataplex-datascan tests
ludoo Aug 17, 2023
9ee0f68
net-vpc
ludoo Aug 17, 2023
78734de
net-vpc test examples
ludoo Aug 17, 2023
a4e9b5d
iam-service-account
ludoo Aug 17, 2023
2a276a2
iam-service-account test examples
ludoo Aug 17, 2023
f5871dd
kms
ludoo Aug 17, 2023
9a515f5
boilerplate
ludoo Aug 17, 2023
8b29a8c
Merge branch 'master' into ludo/modules-iam-design
ludoo Aug 17, 2023
a9d6edc
tfdoc
ludoo Aug 17, 2023
5eb12c5
Merge branch 'ludo/modules-iam-design' of github.com:GoogleCloudPlatf…
ludoo Aug 17, 2023
7b11368
fix module tests
ludoo Aug 17, 2023
46828ff
more blueprint fixes
ludoo Aug 17, 2023
f36fcf9
fix typo in data blueprints
ludoo Aug 17, 2023
c92d839
incomplete refactor of data platform foundations
ludoo Aug 18, 2023
f717b99
tfdoc
ludoo Aug 18, 2023
10d00ad
Merge branch 'master' into ludo/modules-iam-design
ludoo Aug 18, 2023
a29c4ae
data platform foundation
ludoo Aug 18, 2023
a6c0506
refactor data platform foundation iam locals
ludoo Aug 18, 2023
321b3a4
remove redundant example test
ludoo Aug 18, 2023
983581b
shielded folder fix
ludoo Aug 18, 2023
0260ce9
fix typo
ludoo Aug 18, 2023
6b1a39d
project factory
ludoo Aug 18, 2023
ed01414
Merge branch 'master' into ludo/modules-iam-design
ludoo Aug 18, 2023
21d4dc9
project factory outputs
ludoo Aug 18, 2023
d2398c1
Merge branch 'master' into ludo/modules-iam-design
ludoo Aug 18, 2023
0763f26
tfdoc
ludoo Aug 18, 2023
ad582bb
test workflow: less verbose tests, fix tf version
ludoo Aug 19, 2023
69fd2cb
re-enable -vv, shorter traceback, fix action version
ludoo Aug 19, 2023
6193208
ignore github extension warning, re-enable action version
ludoo Aug 19, 2023
4f3c1e4
fast bootstrap IAM, untested
ludoo Aug 19, 2023
24dfadb
bootstrap stage IAM fixes
ludoo Aug 19, 2023
dd6a5a1
stage 0 tests
ludoo Aug 19, 2023
1367330
fast stage 1
ludoo Aug 19, 2023
118bbe2
tenant stage 1
ludoo Aug 19, 2023
fc86ac3
minor changes to fast stage 0 and 1
ludoo Aug 19, 2023
a787cb3
fast security stage
ludoo Aug 19, 2023
30f440b
fast mt stage 0
ludoo Aug 19, 2023
9884b9d
fast mt stage 0
ludoo Aug 19, 2023
b5e3893
fast pf
ludoo Aug 19, 2023
6f5ef65
Update 20230816-iam-refactor.md
ludoo Aug 20, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/daily-tag.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ jobs:
name: "Create tag on master if there was activity in last 24 hours"
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3

- name: "Check changes and tag"
run: |
Expand Down
3 changes: 0 additions & 3 deletions .github/workflows/linting.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,6 @@ on:
pull_request:
branches:
- master
tags:
- ci
- lint

jobs:
linting:
Expand Down
25 changes: 11 additions & 14 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,6 @@ on:
pull_request:
branches:
- master
tags:
- ci
- test

env:
GOOGLE_APPLICATION_CREDENTIALS: "/home/runner/credentials.json"
Expand All @@ -39,7 +36,7 @@ jobs:

- uses: hashicorp/setup-terraform@v2
with:
terraform_version: ${{ env.TERRAFORM_VERSION }}
terraform_version: ${{ env.TF_VERSION }}
terraform_wrapper: false

- name: Build lockfile and fetch providers
Expand Down Expand Up @@ -76,10 +73,10 @@ jobs:
uses: ./.github/actions/fabric-tests
with:
PYTHON_VERSION: ${{ env.PYTHON_VERSION }}
TERRAFORM_VERSION: ${{ env.TERRAFORM_VERSION }}
TERRAFORM_VERSION: ${{ env.TF_VERSION }}

- name: Run tests on documentation examples
run: pytest -vv -n4 -k blueprints/ tests/examples
run: pytest -vv -n4 --tb=line -k blueprints/ tests/examples

examples-modules:
runs-on: ubuntu-latest
Expand All @@ -91,10 +88,10 @@ jobs:
uses: ./.github/actions/fabric-tests
with:
PYTHON_VERSION: ${{ env.PYTHON_VERSION }}
TERRAFORM_VERSION: ${{ env.TERRAFORM_VERSION }}
TERRAFORM_VERSION: ${{ env.TF_VERSION }}

- name: Run tests on documentation examples
run: pytest -vv -n4 -k modules/ tests/examples
run: pytest -vv -n4 --tb=line -k modules/ tests/examples

blueprints:
runs-on: ubuntu-latest
Expand All @@ -106,10 +103,10 @@ jobs:
uses: ./.github/actions/fabric-tests
with:
PYTHON_VERSION: ${{ env.PYTHON_VERSION }}
TERRAFORM_VERSION: ${{ env.TERRAFORM_VERSION }}
TERRAFORM_VERSION: ${{ env.TF_VERSION }}

- name: Run tests environments
run: pytest -vv -n4 tests/blueprints
run: pytest -vv -n4 --tb=line tests/blueprints

modules:
runs-on: ubuntu-latest
Expand All @@ -121,10 +118,10 @@ jobs:
uses: ./.github/actions/fabric-tests
with:
PYTHON_VERSION: ${{ env.PYTHON_VERSION }}
TERRAFORM_VERSION: ${{ env.TERRAFORM_VERSION }}
TERRAFORM_VERSION: ${{ env.TF_VERSION }}

- name: Run tests modules
run: pytest -vv -n4 tests/modules
run: pytest -vv -n4 --tb=line tests/modules

fast:
runs-on: ubuntu-latest
Expand All @@ -136,7 +133,7 @@ jobs:
uses: ./.github/actions/fabric-tests
with:
PYTHON_VERSION: ${{ env.PYTHON_VERSION }}
TERRAFORM_VERSION: ${{ env.TERRAFORM_VERSION }}
TERRAFORM_VERSION: ${{ env.TF_VERSION }}

- name: Run tests on FAST stages
run: pytest -vv -n4 tests/fast
run: pytest -vv -n4 --tb=line tests/fast
Original file line number Diff line number Diff line change
Expand Up @@ -13,21 +13,20 @@ This is the high level diagram:
This sample creates\updates several distinct groups of resources:

- projects
- Deploy M4CE host project with [required services](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#enabling_required_services_on_the_host_project) on a new or existing project.
- M4CE target project prerequisites deployed on existing projects.
- Deploy M4CE host project with [required services](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#enabling_required_services_on_the_host_project) on a new or existing project.
- M4CE target project prerequisites deployed on existing projects.
- IAM
- Create a [service account](https://cloud.google.com/migrate/virtual-machines/docs/5.0/how-to/migrate-connector#step-3) used at runtime by the M4CE connector for data replication
- Grant [migration admin roles](https://cloud.google.com/migrate/virtual-machines/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user accounts
- Grant [migration viewer role](https://cloud.google.com/migrate/virtual-machines/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user accounts
- Grant [migration admin roles](https://cloud.google.com/migrate/virtual-machines/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user or group
- Grant [migration viewer role](https://cloud.google.com/migrate/virtual-machines/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user or group
<!-- BEGIN TFDOC -->

## Variables

| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [migration_admin_users](variables.tf#L15) | List of users authorized to create a new M4CE sources and perform all other migration operations, in IAM format. | <code>list&#40;string&#41;</code> | ✓ | |
| [migration_admin](variables.tf#L15) | User or group who can create a new M4CE sources and perform all other migration operations, in IAM format (`group:[email protected]`). | <code>string</code> | ✓ | |
| [migration_target_projects](variables.tf#L20) | List of target projects for m4ce workload migrations. | <code>list&#40;string&#41;</code> | ✓ | |
| [migration_viewer_users](variables.tf#L25) | List of users authorized to retrieve information about M4CE in the Google Cloud Console, in IAM format. | <code>list&#40;string&#41;</code> | | <code>&#91;&#93;</code> |
| [migration_viewer](variables.tf#L25) | User or group authorized to retrieve information about M4CE in the Google Cloud Console, in IAM format (`group:[email protected]`). | <code>string</code> | | <code>null</code> |
| [project_create](variables.tf#L31) | Parameters for the creation of the new project to host the M4CE backend. | <code title="object&#40;&#123;&#10; billing_account_id &#61; string&#10; parent &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [project_name](variables.tf#L40) | Name of an existing project or of the new project assigned as M4CE host project. | <code>string</code> | | <code>&#34;m4ce-host-project-000&#34;</code> |

Expand All @@ -36,9 +35,7 @@ This sample creates\updates several distinct groups of resources:
| name | description | sensitive |
|---|---|:---:|
| [m4ce_gmanaged_service_account](outputs.tf#L15) | Google managed service account created automatically during the migrate connector registration.. It is used by M4CE to perform activities on target projects. | |

<!-- END TFDOC -->

## Test

```hcl
Expand All @@ -48,8 +45,8 @@ module "test" {
billing_account_id = "1234-ABCD-1234"
parent = "folders/1234563"
}
migration_admin_users = ["user:[email protected]"]
migration_viewer_users = ["user:[email protected]"]
migration_admin = "user:[email protected]"
migration_viewer = "user:[email protected]"
migration_target_projects = [module.test-target-project.name]
depends_on = [
module.test-target-project
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@ module "host-project" {
: null
)
name = var.project_name
parent = (var.project_create != null
parent = (
var.project_create != null
? var.project_create.parent
: null
)

services = [
"cloudresourcemanager.googleapis.com",
"compute.googleapis.com",
Expand All @@ -33,14 +33,24 @@ module "host-project" {
"servicecontrol.googleapis.com",
"vmmigration.googleapis.com",
]

project_create = var.project_create != null

iam_additive = {
"roles/iam.serviceAccountKeyAdmin" = var.migration_admin_users,
"roles/iam.serviceAccountCreator" = var.migration_admin_users,
"roles/vmmigration.admin" = var.migration_admin_users,
"roles/vmmigration.viewer" = var.migration_viewer_users,
iam_bindings_additive = {
admin_sa_key_admin = {
role = "roles/iam.serviceAccountKeyAdmin"
member = var.migration_admin
}
admin_sa_creator = {
role = "roles/iam.serviceAccountCreator"
member = var.migration_admin
}
admin_vmm_admin = {
role = "roles/vmmigration.admin"
member = var.migration_admin
}
viewer_vmm_viewer = {
role = "roles/vmmigration.viewer"
member = var.migration_viewer
}
}
}

Expand All @@ -56,18 +66,25 @@ module "target-projects" {
source = "../../../../modules/project"
name = each.key
project_create = false

services = [
"servicemanagement.googleapis.com",
"servicecontrol.googleapis.com",
"iam.googleapis.com",
"cloudresourcemanager.googleapis.com",
"compute.googleapis.com"
]

iam_additive = {
"roles/resourcemanager.projectIamAdmin" = var.migration_admin_users,
"roles/compute.viewer" = var.migration_admin_users,
"roles/iam.serviceAccountUser" = var.migration_admin_users
iam_bindings_additive = {
admin_project_iam_admin = {
role = "roles/resourcemanager.projectIamAdmin"
member = var.migration_admin
}
admin_compute_viewer = {
role = "roles/compute.viewer"
member = var.migration_admin
}
admin_sa_user = {
role = "roles/iam.serviceAccountUser"
member = var.migration_admin
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -12,20 +12,20 @@
# See the License for the specific language governing permissions and
# limitations under the License.

variable "migration_admin_users" {
description = "List of users authorized to create a new M4CE sources and perform all other migration operations, in IAM format."
type = list(string)
variable "migration_admin" {
description = "User or group who can create a new M4CE sources and perform all other migration operations, in IAM format (`group:[email protected]`)."
type = string
}

variable "migration_target_projects" {
description = "List of target projects for m4ce workload migrations."
type = list(string)
}

variable "migration_viewer_users" {
description = "List of users authorized to retrieve information about M4CE in the Google Cloud Console, in IAM format."
type = list(string)
default = []
variable "migration_viewer" {
description = "User or group authorized to retrieve information about M4CE in the Google Cloud Console, in IAM format (`group:[email protected]`)."
type = string
default = null
}

variable "project_create" {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,34 +13,33 @@ This is the high level diagram:
This sample creates\update several distinct groups of resources:

- projects
- M4CE host project with [required services](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#enabling_required_services_on_the_host_project) deployed on a new or existing project.
- M4CE target project prerequisites deployed on existing projects.
- M4CE host project with [required services](https://cloud.google.com/migrate/compute-engine/docs/5.0/how-to/enable-services#enabling_required_services_on_the_host_project) deployed on a new or existing project.
- M4CE target project prerequisites deployed on existing projects.
- IAM
- Create a [service account](https://cloud.google.com/migrate/virtual-machines/docs/5.0/how-to/migrate-connector#step-3) used at runtime by the M4CE connector for data replication
- Grant [migration admin roles](https://cloud.google.com/migrate/virtual-machines/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user accounts.
- Grant [migration viewer role](https://cloud.google.com/migrate/virtual-machines/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user accounts.
- Grant [roles on shared VPC](https://cloud.google.com/migrate/virtual-machines/docs/5.0/how-to/target-project#configure-permissions) to migration admins
- Grant [migration admin roles](https://cloud.google.com/migrate/virtual-machines/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user or group.
- Grant [migration viewer role](https://cloud.google.com/migrate/virtual-machines/docs/5.0/how-to/enable-services#using_predefined_roles) to provided user or group.
- Grant [roles on shared VPC](https://cloud.google.com/migrate/virtual-machines/docs/5.0/how-to/target-project#configure-permissions) to migration user or group
<!-- BEGIN TFDOC -->

## Variables

| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [migration_admin_users](variables.tf#L15) | List of users authorized to create a new M4CE sources and perform all other migration operations, in IAM format. | <code>list&#40;string&#41;</code> | ✓ | |
| [migration_admin](variables.tf#L15) | User or group who can create a new M4CE sources and perform all other migration operations, in IAM format (`group:[email protected]`). | <code>string</code> | ✓ | |
| [migration_target_projects](variables.tf#L20) | List of target projects for m4ce workload migrations. | <code>list&#40;string&#41;</code> | ✓ | |
| [sharedvpc_host_projects](variables.tf#L45) | List of host projects that share a VPC with the selected target projects. | <code>list&#40;string&#41;</code> | ✓ | |
| [migration_viewer_users](variables.tf#L25) | List of users authorized to retrieve information about M4CE in the Google Cloud Console, in IAM format. | <code>list&#40;string&#41;</code> | | <code>&#91;&#93;</code> |
| [project_create](variables.tf#L30) | Parameters for the creation of the new project to host the M4CE backend. | <code title="object&#40;&#123;&#10; billing_account_id &#61; string&#10; parent &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [project_name](variables.tf#L39) | Name of an existing project or of the new project assigned as M4CE host project. | <code>string</code> | | <code>&#34;m4ce-host-project-000&#34;</code> |
| [sharedvpc_host_projects](variables.tf#L46) | List of host projects that share a VPC with the selected target projects. | <code>list&#40;string&#41;</code> | ✓ | |
| [migration_viewer](variables.tf#L25) | User or group authorized to retrieve information about M4CE in the Google Cloud Console, in IAM format (`group:[email protected]`). | <code>string</code> | | <code>null</code> |
| [project_create](variables.tf#L31) | Parameters for the creation of the new project to host the M4CE backend. | <code title="object&#40;&#123;&#10; billing_account_id &#61; string&#10; parent &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [project_name](variables.tf#L40) | Name of an existing project or of the new project assigned as M4CE host project. | <code>string</code> | | <code>&#34;m4ce-host-project-000&#34;</code> |

## Outputs

| name | description | sensitive |
|---|---|:---:|
| [m4ce_gmanaged_service_account](outputs.tf#L15) | Google managed service account created automatically during the migrate connector registration. It is used by M4CE to perform activities on target projects. | |

<!-- END TFDOC -->
## Manual Steps

Once this blueprint is deployed the M4CE [m4ce_gmanaged_service_account](https://cloud.google.com/migrate/virtual-machines/docs/5.0/how-to/target-sa-compute-engine#configuring_the_default_service_account) has to be configured to grant the access to the shared VPC and allow the deploy of Compute Engine instances as the result of the migration.

## Test
Expand All @@ -52,8 +51,8 @@ module "test" {
billing_account_id = "1234-ABCD-1234"
parent = "folders/1234563"
}
migration_admin_users = ["user:[email protected]"]
migration_viewer_users = ["user:[email protected]"]
migration_admin = "user:[email protected]"
migration_viewer = "user:[email protected]"
migration_target_projects = [module.test-target-project.name]
sharedvpc_host_projects = [module.test-sharedvpc-host-project.name]
depends_on = [
Expand Down
Loading