From d89e50d8d5c7fd68f29c5d192e58ae1dd5f91b50 Mon Sep 17 00:00:00 2001 From: Sam Levenick Date: Fri, 12 Nov 2021 16:32:38 -0800 Subject: [PATCH 1/4] Add a guide on authoring tpgtools new resources --- tpgtools/README.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/tpgtools/README.md b/tpgtools/README.md index c3e2b7561bf6..27c27dcaa3fc 100644 --- a/tpgtools/README.md +++ b/tpgtools/README.md @@ -176,3 +176,33 @@ doc_hide: test_hide: - basic_workload.yaml ``` + +#### New Resource Guide (tpgtools only) + +This guide is written to document the process for adding a resource to the Google Terraform Provider (TPG) after it has been added to the [DCL](https://github.com/GoogleCloudPlatform/declarative-resource-client-library). + +## Adding Resource Overrides + +Every resource added via tpgtools needs an override file for every version it is available at. This file may be empty, but must exist. A resource available at GA (TPG) must also exist at beta (TPGB) and needs a corresponding override file at beta. These override files are often identical between versions. This file should exist at tpgtools/overrides/$PRODUCT_NAME/$VERSION/$RESOURCE.yaml. For example, [this override](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/tpgtools/overrides/assuredworkloads/beta/workload.yaml) exists for the product assuredworkloads resource workload at beta version. + +# What should I add to the overrides file? + +For the first pass try nothing. If behavior is needed to change the resource's behavior in Terraform from the default DCL resource, then you can add these at a later date. Generally no overrides will be needed for a new resource, but they will be needed for resources that exist in mmv1 that are being converted to use tpgtools. + +## Adding Samples + +See [guide above](#samples) for adding samples. + +## Adding Additional Information to Handwritten/MMv1 Generated Files + +There are a few files within mmv1 that use Ruby templates to generate shared files for the provider. One of these is provider.go (generated by [provider.go.erb](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/mmv1/third_party/terraform/utils/provider.go.erb)) which contains the map of resources to their names within the provider. Currently these need to be updated by hand to add a new resource. There are a couple of other places that need to be modified as well to add endpoint information if the resource being added is part of a new product. + +# Adding to provider.go + +Add the resource definition within the `ResourceMapWithErrors` map. There is currently a block of tpgtools resources, so adding to that block is preferred. If this resource is available only at a specific version (beta, private) then add version tags around the resource definition. + +Additionally you may need to add a few line to provider.go for the endpoint if this resource is in a new product that doesn't have any other resources currently in the provider. Find the correct lines in the diff on `provider_dcl_endpoints.go` within the provider once it has been generated with the new resource. + +## Adding Documentation + +Documentation will be automatically generated based on the resource description in the DCL and examples will be generated from the samples for this resource. \ No newline at end of file From 49e98f9caad9f3fa505082972a764da8fb580290 Mon Sep 17 00:00:00 2001 From: Sam Levenick Date: Fri, 12 Nov 2021 16:34:25 -0800 Subject: [PATCH 2/4] Wrong headers --- tpgtools/README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/tpgtools/README.md b/tpgtools/README.md index 27c27dcaa3fc..0b3431e1f027 100644 --- a/tpgtools/README.md +++ b/tpgtools/README.md @@ -177,32 +177,32 @@ test_hide: - basic_workload.yaml ``` -#### New Resource Guide (tpgtools only) +## New Resource Guide (tpgtools only) This guide is written to document the process for adding a resource to the Google Terraform Provider (TPG) after it has been added to the [DCL](https://github.com/GoogleCloudPlatform/declarative-resource-client-library). -## Adding Resource Overrides +### Adding Resource Overrides Every resource added via tpgtools needs an override file for every version it is available at. This file may be empty, but must exist. A resource available at GA (TPG) must also exist at beta (TPGB) and needs a corresponding override file at beta. These override files are often identical between versions. This file should exist at tpgtools/overrides/$PRODUCT_NAME/$VERSION/$RESOURCE.yaml. For example, [this override](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/tpgtools/overrides/assuredworkloads/beta/workload.yaml) exists for the product assuredworkloads resource workload at beta version. -# What should I add to the overrides file? +#### What should I add to the overrides file? For the first pass try nothing. If behavior is needed to change the resource's behavior in Terraform from the default DCL resource, then you can add these at a later date. Generally no overrides will be needed for a new resource, but they will be needed for resources that exist in mmv1 that are being converted to use tpgtools. -## Adding Samples +### Adding Samples See [guide above](#samples) for adding samples. -## Adding Additional Information to Handwritten/MMv1 Generated Files +### Adding Additional Information to Handwritten/MMv1 Generated Files There are a few files within mmv1 that use Ruby templates to generate shared files for the provider. One of these is provider.go (generated by [provider.go.erb](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/mmv1/third_party/terraform/utils/provider.go.erb)) which contains the map of resources to their names within the provider. Currently these need to be updated by hand to add a new resource. There are a couple of other places that need to be modified as well to add endpoint information if the resource being added is part of a new product. -# Adding to provider.go +#### Adding to provider.go Add the resource definition within the `ResourceMapWithErrors` map. There is currently a block of tpgtools resources, so adding to that block is preferred. If this resource is available only at a specific version (beta, private) then add version tags around the resource definition. Additionally you may need to add a few line to provider.go for the endpoint if this resource is in a new product that doesn't have any other resources currently in the provider. Find the correct lines in the diff on `provider_dcl_endpoints.go` within the provider once it has been generated with the new resource. -## Adding Documentation +### Adding Documentation -Documentation will be automatically generated based on the resource description in the DCL and examples will be generated from the samples for this resource. \ No newline at end of file +Documentation will be automatically generated based on the resource description in the DCL and examples will be generated from the samples for this resource. From c15f18a08501ed984ccd9b27ab859cfc2e221e6f Mon Sep 17 00:00:00 2001 From: Sam Levenick Date: Fri, 19 Nov 2021 11:49:24 -0800 Subject: [PATCH 3/4] Simplify samples, reorder per PR feedback --- tpgtools/README.md | 145 ++++++++++++++++----------------------------- 1 file changed, 50 insertions(+), 95 deletions(-) diff --git a/tpgtools/README.md b/tpgtools/README.md index 0b3431e1f027..b884238faa8b 100644 --- a/tpgtools/README.md +++ b/tpgtools/README.md @@ -51,39 +51,24 @@ To generate accessory code such as `serializarion`, you can specify the `--mode` go run . --path "api" --overrides "overrides" --output ~/some/dir --mode "serialization" ``` -## Development - -`tpgtools` builds resources using Go Templates, with the templates stored under -the `templates/` directory. They're fed the `Resource` type, which contains -resource-level metadata and a list of `Property` types which represent top-level -fields in a Terraform resource. Additionally, `Property`s can contain other -`Property` objects to create a nested structure of fields. +## New Resource Guide -`main.go` parses the OpenAPI schemas you've selected into Go structs and then -transforms them into `Resource`/`Property` before running them through Go -Templates. +This guide is written to document the process for adding a resource to the Google Terraform Provider (TPG) after it has been added to the [DCL](https://github.com/GoogleCloudPlatform/declarative-resource-client-library). -### Overrides +### Adding Resource Overrides -Overrides are specified per-resource, with a directory structure parallel to the -OpenAPI specs. Inside each resource file is an unordered array of overrides made -up of an override type (like `CUSTOM_DESCRIPTION` or `VIRTUAL_FIELD`) as well as -a field they affect (if a field is omitted, they affect the resource) and an -optional `details` object that will be parsed for additional metadata. +Every resource added via tpgtools needs an override file for every version it is available at. This file should be empty, but must exist. A resource available at GA (TPG) must also exist at beta (TPGB) and needs a corresponding override file at beta. These override files are often identical between versions. This file should exist at tpgtools/overrides/$PRODUCT_NAME/$VERSION/$RESOURCE.yaml. For example, [this override](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/tpgtools/overrides/assuredworkloads/beta/workload.yaml) exists for the product assuredworkloads resource workload at beta version. -For example, override entries will look like the following: +Override files contain information on how the Terraform representation of the resource should be different from the DCL's representation. This could be naming or behavior differences, but for a new resource implemented through tpgtools there should be no differences from the DCL's representation. -```yaml -- type: CUSTOM_DESCRIPTION - field: lifecycle.rule.condition.age - details: - description: Custom description here. -``` +### Adding Samples -#### Samples +Adding samples is essential for generating tests and documentation. -We will autoingest samples from the dcl, however we currently must -manually fill the substitutions for these samples. +Start by copying the relevant samples from the DCL for your new resource. These will be added to +the [tpgtools/api folder](https://github.com/GoogleCloudPlatform/magic-modules/tree/master/tpgtools/api) under the relevant product. +These samples can be found under the samples/ folder within the DCL for the resource being added. For example, assured +workloads can be found [here](https://github.com/GoogleCloudPlatform/declarative-resource-client-library/tree/main/services/google/assuredworkloads/samples). You may need to re-serialize `serialization.go` if you are adding newer resources. To do this @@ -93,69 +78,22 @@ go get -u github.com/GoogleCloudPlatform/declarative-resource-client-library make serialize ``` -To do so, first create a folder in the relevant product -``` -$ cd overrides/{{product}}/ -$ mkdir samples -$ cd samples -``` +Create a meta.yaml file in the overrides directory for the resource. For example, +[this meta.yaml](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/tpgtools/overrides/assuredworkloads/samples/workload/meta.yaml) file exists for the assured workloads resource. +This file will merge with any tests you customize behavior +of the tests and examples generated from the samples data (injections, hiding, ect..) -Then make a folder for the resource you would like to add samples for. -``` -$ mkdir {{resource}} -$ cd resource -``` - -Create a meta.yaml file. This file will merge with any tests you create -providing sustitutions and other relevant test data (injections, hiding, ect..) - -Provide the relevant sustitutions needed. See the referenced variables in the dcl -jsons. They should surrounded by `{{}}` -``` -substitutions: - - substitution: "project" - value: ":PROJECT" - - substitution: "region" - value: ":REGION" - - substitution: "name" - value: "trigger" - - substitution: "topic" - value: "topic" - - substitution: "event_arc_service" - value: "service-eventarc" - - substitution: "service_account" - value: "sa" -``` - -If you need to hide sample from doc or hide a sample from docs you can do so here as well. +If you need to hide sample from doc or hide a sample from docs you can do so here. ``` doc_hide: - basic.tf.tmpl + - full.tf.tmpl test_hide: - - basic_trigger.yaml -``` - -Any files with a `.tf.tmpl` (terraform template) extension located in the `override/{{product}}samples/{{resource}}` directory -and without `_update` in the name are considered to be tests independently. -These are normal terraform files with the desired sustituted variables surrounded by `{{}}`. -These tests will also use the substitutions defined in the `meta.yaml`. If you want to provide test specific -rules (updates, ect), you can create a yaml file with the same name as the `.tf.tmpl` file. Here you can supply updates -or version requirements specific to this test. If version is ommitted the sample assumed to run against all versions. -``` -updates: - - resource: basic_update_transport.tf.tmpl - - resource: basic_update_transport_2.tf.tmpl -version: - - beta + - basic_workload.yaml ``` If you want to add additional rules the following options are currently supported within `meta.yaml` ``` - - substitution: "name" - value: "workload" - - substitution: "region" - value: ":REGION" - .... ignore_read: - "billing_account" - "kms_settings" @@ -177,32 +115,49 @@ test_hide: - basic_workload.yaml ``` -## New Resource Guide (tpgtools only) +### Registering New Resources -This guide is written to document the process for adding a resource to the Google Terraform Provider (TPG) after it has been added to the [DCL](https://github.com/GoogleCloudPlatform/declarative-resource-client-library). +There are a few files within mmv1 that use Ruby templates to generate shared files for the provider. One of these is provider.go (generated by [provider.go.erb](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/mmv1/third_party/terraform/utils/provider.go.erb)) which contains the map of resources to their names within the provider. Currently these need to be updated by hand to add a new resource. There are a couple of other places that need to be modified as well to add endpoint information if the resource being added is part of a new product. -### Adding Resource Overrides +#### Adding to provider.go -Every resource added via tpgtools needs an override file for every version it is available at. This file may be empty, but must exist. A resource available at GA (TPG) must also exist at beta (TPGB) and needs a corresponding override file at beta. These override files are often identical between versions. This file should exist at tpgtools/overrides/$PRODUCT_NAME/$VERSION/$RESOURCE.yaml. For example, [this override](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/tpgtools/overrides/assuredworkloads/beta/workload.yaml) exists for the product assuredworkloads resource workload at beta version. +Add the resource definition within the `ResourceMapWithErrors` map. There is currently a block of tpgtools resources, so adding to that block is preferred. If this resource is available only at a specific version (beta, private) then add version tags around the resource definition. -#### What should I add to the overrides file? +Additionally you may need to add a few line to provider.go for the endpoint if this resource is in a new product that doesn't have any other resources currently in the provider. Find the correct lines in the diff on `provider_dcl_endpoints.go` within the provider once it has been generated with the new resource. -For the first pass try nothing. If behavior is needed to change the resource's behavior in Terraform from the default DCL resource, then you can add these at a later date. Generally no overrides will be needed for a new resource, but they will be needed for resources that exist in mmv1 that are being converted to use tpgtools. +### Adding Documentation -### Adding Samples +Provided you have added samples for the resource, documentation will be automatically generated based on the resource description in the DCL and examples will be generated from the samples for this resource. If you did not provide samples for the resource, then documentation will need to be written by hand. -See [guide above](#samples) for adding samples. +### Handwritten Tests -### Adding Additional Information to Handwritten/MMv1 Generated Files +Sometimes you may need to test unusual resource behavior in a way that does not fit well with generated tests. In this circumstance you can write a test file and add it [here](https://github.com/GoogleCloudPlatform/magic-modules/tree/master/mmv1/third_party/terraform/tests). These tests can be used for more granular testing of specific behavior and add custom checks. Tests in these files will not have examples generated for them, so handwritten tests should not be considered a replacement for samples. -There are a few files within mmv1 that use Ruby templates to generate shared files for the provider. One of these is provider.go (generated by [provider.go.erb](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/mmv1/third_party/terraform/utils/provider.go.erb)) which contains the map of resources to their names within the provider. Currently these need to be updated by hand to add a new resource. There are a couple of other places that need to be modified as well to add endpoint information if the resource being added is part of a new product. +## Development -#### Adding to provider.go +`tpgtools` builds resources using Go Templates, with the templates stored under +the `templates/` directory. They're fed the `Resource` type, which contains +resource-level metadata and a list of `Property` types which represent top-level +fields in a Terraform resource. Additionally, `Property`s can contain other +`Property` objects to create a nested structure of fields. -Add the resource definition within the `ResourceMapWithErrors` map. There is currently a block of tpgtools resources, so adding to that block is preferred. If this resource is available only at a specific version (beta, private) then add version tags around the resource definition. +`main.go` parses the OpenAPI schemas you've selected into Go structs and then +transforms them into `Resource`/`Property` before running them through Go +Templates. -Additionally you may need to add a few line to provider.go for the endpoint if this resource is in a new product that doesn't have any other resources currently in the provider. Find the correct lines in the diff on `provider_dcl_endpoints.go` within the provider once it has been generated with the new resource. +### Overrides -### Adding Documentation +Overrides are specified per-resource, with a directory structure parallel to the +OpenAPI specs. Inside each resource file is an unordered array of overrides made +up of an override type (like `CUSTOM_DESCRIPTION` or `VIRTUAL_FIELD`) as well as +a field they affect (if a field is omitted, they affect the resource) and an +optional `details` object that will be parsed for additional metadata. + +For example, override entries will look like the following: -Documentation will be automatically generated based on the resource description in the DCL and examples will be generated from the samples for this resource. +```yaml +- type: CUSTOM_DESCRIPTION + field: lifecycle.rule.condition.age + details: + description: Custom description here. +``` From 9095a61e1d8f0e8e40c21fefd7919c284195b5ad Mon Sep 17 00:00:00 2001 From: Sam Levenick Date: Fri, 19 Nov 2021 12:51:52 -0800 Subject: [PATCH 4/4] PR feedback --- tpgtools/README.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/tpgtools/README.md b/tpgtools/README.md index b884238faa8b..616ad227e2cb 100644 --- a/tpgtools/README.md +++ b/tpgtools/README.md @@ -70,11 +70,9 @@ the [tpgtools/api folder](https://github.com/GoogleCloudPlatform/magic-modules/t These samples can be found under the samples/ folder within the DCL for the resource being added. For example, assured workloads can be found [here](https://github.com/GoogleCloudPlatform/declarative-resource-client-library/tree/main/services/google/assuredworkloads/samples). -You may need to re-serialize `serialization.go` if you are adding newer resources. -To do this +Re-serialize `serialization.go` to enable generating configs from samples by running: ``` -cd tpgtools -go get -u github.com/GoogleCloudPlatform/declarative-resource-client-library +make upgrade-dcl make serialize ```