Add a guide on authoring tpgtools new resources (GoogleCloudPlatform#…

…5452)

* Add a guide on authoring tpgtools new resources

* Wrong headers

* Simplify samples, reorder per PR feedback

* PR feedback

tpgtools/README.md

@@ -51,111 +51,47 @@ To generate accessory code such as `serializarion`, you can specify the `--mode`
 go run . --path "api" --overrides "overrides" --output ~/some/dir --mode "serialization"
 ```
 
-## Development
+## New Resource Guide
 
-`tpgtools` builds resources using Go Templates, with the templates stored under
-the `templates/` directory. They're fed the `Resource` type, which contains
-resource-level metadata and a list of `Property` types which represent top-level
-fields in a Terraform resource. Additionally, `Property`s can contain other
-`Property` objects to create a nested structure of fields.
+This guide is written to document the process for adding a resource to the Google Terraform Provider (TPG) after it has been added to the [DCL](https://github.com/GoogleCloudPlatform/declarative-resource-client-library).
 
-`main.go` parses the OpenAPI schemas you've selected into Go structs and then
-transforms them into `Resource`/`Property` before running them through Go
-Templates.
+### Adding Resource Overrides
 
-### Overrides
+Every resource added via tpgtools needs an override file for every version it is available at. This file should be empty, but must exist. A resource available at GA (TPG) must also exist at beta (TPGB) and needs a corresponding override file at beta. These override files are often identical between versions. This file should exist at tpgtools/overrides/$PRODUCT_NAME/$VERSION/$RESOURCE.yaml. For example, [this override](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/tpgtools/overrides/assuredworkloads/beta/workload.yaml) exists for the product assuredworkloads resource workload at beta version.
 
-Overrides are specified per-resource, with a directory structure parallel to the
-OpenAPI specs. Inside each resource file is an unordered array of overrides made
-up of an override type (like `CUSTOM_DESCRIPTION` or `VIRTUAL_FIELD`) as well as
-a field they affect (if a field is omitted, they affect the resource) and an
-optional `details` object that will be parsed for additional metadata.
+Override files contain information on how the Terraform representation of the resource should be different from the DCL's representation. This could be naming or behavior differences, but for a new resource implemented through tpgtools there should be no differences from the DCL's representation.
 
-For example, override entries will look like the following:
-
-```yaml
-- type: CUSTOM_DESCRIPTION
-  field: lifecycle.rule.condition.age
-  details:
-    description: Custom description here.
-```
+### Adding Samples
 
-#### Samples
+Adding samples is essential for generating tests and documentation.
 
-We will autoingest samples from the dcl, however we currently must
-manually fill the substitutions for these samples.
+Start by copying the relevant samples from the DCL for your new resource. These will be added to
+the [tpgtools/api folder](https://github.com/GoogleCloudPlatform/magic-modules/tree/master/tpgtools/api) under the relevant product.
+These samples can be found under the samples/ folder within the DCL for the resource being added. For example, assured
+workloads can be found [here](https://github.com/GoogleCloudPlatform/declarative-resource-client-library/tree/main/services/google/assuredworkloads/samples).
 
-You may need to re-serialize `serialization.go` if you are adding newer resources.
-To do this
+Re-serialize `serialization.go` to enable generating configs from samples by running:
 ```
-cd tpgtools
-go get -u github.com/GoogleCloudPlatform/declarative-resource-client-library
+make upgrade-dcl
 make serialize
 ```
 
-To do so, first create a folder in the relevant product
-```
-$ cd overrides/{{product}}/
-$ mkdir samples
-$ cd samples
-```
-
-Then make a folder for the resource you would like to add samples for.
-```
-$ mkdir {{resource}}
-$ cd resource
-```
-
-Create a meta.yaml file. This file will merge with any tests you create
-providing sustitutions and other relevant test data (injections, hiding, ect..)
+Create a meta.yaml file in the overrides directory for the resource. For example, 
+[this meta.yaml](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/tpgtools/overrides/assuredworkloads/samples/workload/meta.yaml) file exists for the assured workloads resource.
+This file will merge with any tests you customize behavior
+of the tests and examples generated from the samples data (injections, hiding, ect..)
 
-Provide the relevant sustitutions needed. See the referenced variables in the dcl
-jsons. They should surrounded by `{{}}`
-```
-substitutions:
-  - substitution: "project"
-    value: ":PROJECT"
-  - substitution: "region"
-    value: ":REGION"
-  - substitution: "name"
-    value: "trigger"
-  - substitution: "topic"
-    value: "topic"
-  - substitution: "event_arc_service"
-    value: "service-eventarc"
-  - substitution: "service_account"
-    value: "sa"
-```
-
-If you need to hide sample from doc or hide a sample from docs you can do so here as well.
+If you need to hide sample from doc or hide a sample from docs you can do so here.
 ```
 doc_hide:
   - basic.tf.tmpl
+  - full.tf.tmpl
 test_hide:
-  - basic_trigger.yaml
-```
-
-Any files with a `.tf.tmpl` (terraform template) extension located in the `override/{{product}}samples/{{resource}}` directory
-and without `_update` in the name are considered to be tests independently.
-These are normal terraform files with the desired sustituted variables surrounded by `{{}}`.
-These tests will also use the substitutions defined in the `meta.yaml`. If you want to provide test specific
-rules (updates, ect), you can create a yaml file with the same name as the `.tf.tmpl` file. Here you can supply updates
-or version requirements specific to this test. If version is ommitted the sample assumed to run against all versions.
-```
-updates:
-  - resource: basic_update_transport.tf.tmpl
-  - resource: basic_update_transport_2.tf.tmpl
-version:
-  - beta
+  - basic_workload.yaml
 ```
 
 If you want to add additional rules the following options are currently supported within `meta.yaml`
 ```
-  - substitution: "name"
-    value: "workload"
-  - substitution: "region"
-    value: ":REGION"
- ....
 ignore_read:
   - "billing_account"
   - "kms_settings"
@@ -176,3 +112,50 @@ doc_hide:
 test_hide:
   - basic_workload.yaml
 ```
+
+### Registering New Resources
+
+There are a few files within mmv1 that use Ruby templates to generate shared files for the provider. One of these is provider.go (generated by [provider.go.erb](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/mmv1/third_party/terraform/utils/provider.go.erb)) which contains the map of resources to their names within the provider. Currently these need to be updated by hand to add a new resource. There are a couple of other places that need to be modified as well to add endpoint information if the resource being added is part of a new product.
+
+#### Adding to provider.go
+
+Add the resource definition within the `ResourceMapWithErrors` map. There is currently a block of tpgtools resources, so adding to that block is preferred. If this resource is available only at a specific version (beta, private) then add version tags around the resource definition.
+
+Additionally you may need to add a few line to provider.go for the endpoint if this resource is in a new product that doesn't have any other resources currently in the provider. Find the correct lines in the diff on `provider_dcl_endpoints.go` within the provider once it has been generated with the new resource.
+
+### Adding Documentation
+
+Provided you have added samples for the resource, documentation will be automatically generated based on the resource description in the DCL and examples will be generated from the samples for this resource. If you did not provide samples for the resource, then documentation will need to be written by hand.
+
+### Handwritten Tests
+
+Sometimes you may need to test unusual resource behavior in a way that does not fit well with generated tests. In this circumstance you can write a test file and add it [here](https://github.com/GoogleCloudPlatform/magic-modules/tree/master/mmv1/third_party/terraform/tests). These tests can be used for more granular testing of specific behavior and add custom checks. Tests in these files will not have examples generated for them, so handwritten tests should not be considered a replacement for samples. 
+
+## Development
+
+`tpgtools` builds resources using Go Templates, with the templates stored under
+the `templates/` directory. They're fed the `Resource` type, which contains
+resource-level metadata and a list of `Property` types which represent top-level
+fields in a Terraform resource. Additionally, `Property`s can contain other
+`Property` objects to create a nested structure of fields.
+
+`main.go` parses the OpenAPI schemas you've selected into Go structs and then
+transforms them into `Resource`/`Property` before running them through Go
+Templates.
+
+### Overrides
+
+Overrides are specified per-resource, with a directory structure parallel to the
+OpenAPI specs. Inside each resource file is an unordered array of overrides made
+up of an override type (like `CUSTOM_DESCRIPTION` or `VIRTUAL_FIELD`) as well as
+a field they affect (if a field is omitted, they affect the resource) and an
+optional `details` object that will be parsed for additional metadata.
+
+For example, override entries will look like the following:
+
+```yaml
+- type: CUSTOM_DESCRIPTION
+  field: lifecycle.rule.condition.age
+  details:
+    description: Custom description here.
+```