add terraform-provider-google-beta to ci #502

danawillow · 2018-09-26T21:39:21Z

This adds all the steps necessary to generate all the resources at their beta version into tpg-beta.

[all]

[terraform]

Output from magician generating into ga and beta providers

[terraform-beta]

[puppet]

[puppet-bigquery]

[puppet-compute]

[puppet-container]

[puppet-dns]

[puppet-logging]

[puppet-pubsub]

[puppet-resourcemanager]

[puppet-sql]

[puppet-storage]

[chef]

[chef-compute]

[chef-container]

[chef-dns]

[chef-logging]

[chef-spanner]

[chef-sql]

[chef-storage]

[ansible]

modular-magician · 2018-10-02T23:09:32Z

I am a robot that works on MagicModules PRs!
I checked the downstream repositories (see README.md for which ones I can write to), and none of them seem to have any changes.

Once this PR is approved, you can feel free to merge it without taking any further steps.

…sh anyway

modular-magician · 2018-10-03T03:17:05Z

I am a robot that works on MagicModules PRs!
I checked the downstream repositories (see README.md for which ones I can write to), and none of them seem to have any changes.

Once this PR is approved, you can feel free to merge it without taking any further steps.

modular-magician · 2018-10-03T16:07:02Z

I am a robot that works on MagicModules PRs!
I checked the downstream repositories (see README.md for which ones I can write to), and none of them seem to have any changes.

Once this PR is approved, you can feel free to merge it without taking any further steps.

danawillow · 2018-10-03T17:24:55Z

The create-prs step failed for unreleated reasons after the PR had already been opened, which I think is why the magician is confused about whether there are any downstream changes. However, it did open hashicorp/terraform-provider-google-beta#2, which looks right.

This is ready for review now, but shouldn't be submitted until we catch up beta with ga, since some of these changes are things that have already been made. We also probaably need to get interconnects to generate into here, since otherwise it's going to keep trying to remove it until that PR is submitted in MM.

nat-henderson · 2018-10-03T19:46:40Z

.ci/magic-modules/create-pr.sh

-      echo "Terraform - did not generate a PR."
-    fi
-    popd
+    for VERSION in '' 'beta'; do


Do we someday plan on an tpg-alpha, or something like that, as well? Basically, I think that either we should have this for-each thing in .ci/ci.yml.tmpl and in here, or else neither, and just duplicate the code below here. I'm inclined towards the first option, but if you think the second is better, or if you think as-written this strikes the right balance, that also makes sense to me.

nat-henderson · 2018-10-03T19:48:36Z

.ci/magic-modules/generate-terraform-beta.yml

@@ -0,0 +1,24 @@
+---
+# This file takes two inputs: magic-modules-branched in detached-HEAD state, and the list of patches.
+# It spits out "terraform-generated", a terraform repo on a new branch (named after the


Comment needs updating. (I wish this file wasn't necessary, ugh, seems like a big pain for no good reason, but it is since the output name changes)

nat-henderson · 2018-10-03T19:49:52Z

.ci/magic-modules/generate-terraform.sh


 # Resources that were already using beta APIs before they started being autogenerated
-bundle exec compiler -v beta -p products/compute -t Address,Firewall,ForwardingRule,GlobalAddress,RegionDisk,Subnetwork,VpnTunnel -e terraform -o "${GOPATH}/src/github.com/terraform-providers/terraform-provider-google/"
+bundle exec compiler -v beta -p products/compute -t Address,Firewall,ForwardingRule,GlobalAddress,RegionDisk,Subnetwork,VpnTunnel -e terraform -o "${GOPATH}/src/github.com/terraform-providers/$PROVIDER_NAME/"

 # This command can crash - if that happens, the script should not fail.
 set +e


This is where you'll need to add --tag terraform-beta if you want the magician to understand differences in pr template tags.

nat-henderson · 2018-10-03T19:50:52Z

.ci/magic-modules/point-to-submodules.sh

-  git submodule sync build/terraform
-  ssh-agent bash -c "ssh-add ~/github_private_key; git submodule update --remote --init build/terraform"
-  git add build/terraform
+  for VERSION in '' 'beta'; do


This is at least as complicated as just doubling up the code below - same comment as above re: duplication vs iteration.

Merged PR #562.

modular-magician · 2018-10-17T23:08:52Z

I am (still) a robot that works on MagicModules PRs!

I just wanted to let you know that your changes (as of commit 0150b36) have been included in your existing downstream PRs.

danawillow · 2018-10-17T23:14:25Z

All right, this is finally ready.

@ndmckinley for CI stuff (and as much tf stuff as you feel like reviewing)
@rileykarson and @chrisst for the rest, including the downstreams.

For the copied-over files, I find it useful to look at the downstreams alongside what's here to see what got generated.

Merged PR #571.

nat-henderson · 2018-10-17T23:29:53Z

.ci/magic-modules/create-pr.sh

-      echo "Terraform - did not generate a PR."
-    fi
-    popd
+    for VERSION in "${TERRAFORM_VERSIONS[@]}"; do


I still kind of think this is a sort of false generality - the best version of generality would be a lookup from version string to provider name. I also kind of think that it's odd to special-case one out of two values here.

just to confirm what this would mean in practice- having a map variable that we can do the PROVIDER_NAME/SUBMODULE_DIR/etc lookups in?

That's the best implementation I can imagine, yeah. As-is, I argue we're very likely to need to change it up if we ever add an tpg-alpha, and a lookup gives us flexibility.

This is great!

) Merged PR #582.

rileykarson

For Terraform only and not CI:

I think everything looks correct - there's no way I can be fully confident in a +- like 7500 line change but if both downstreams build & we run them through the CI gauntlet I'd be happy

Merged PR #575.

chrisst · 2018-10-17T23:28:13Z

provider/terraform/common~compile.yaml

+'<%= dir -%>/<%= fname.split(".erb")[0] -%>': 'provider/terraform/resources/<%= fname -%>'
+<% end -%>
+<%
+  Dir["provider/terraform/data_sources/*.go.erb"].each do |file_path|


without the prefix logic, will this compile every file each time that ./compiler gets called. Once for each product?

Yup. We talked about this a bit in the meeting today, but Emily had realized that we don't have something generate on every api prefix that we would want, so even if we do a compiler -a, we'll still miss some things. Rather than trying to do something super complicated, we'll just compile/copy everything every time until it becomes enough of a nuisance that someone comes up with a good solution :)

chrisst · 2018-10-17T23:33:31Z

templates/terraform/examples/autoscaler_basic.tf.erb

@@ -46,7 +46,14 @@ resource "google_compute_instance_group_manager" "foobar" {
  name = "<%= ctx[:vars]['igm_name'] %>"
  zone = "us-central1-f"

+<% if ctx[:version].nil? || ctx[:version] == 'ga' -%>


Some templates have ctx[:version].nil? and others have if version.nil? is there a reason for the difference?

The examples/generated tests use ctx because that's just how compile_file (the compile command that compiles those in custom_code.rb) works- you pass it a hash of variables that you want the file to have access to. For whatever reason, it's different from compile_files, which is what the other stuff that gets compiled uses.

modular-magician · 2018-10-18T21:55:58Z

I am (still) a robot that works on MagicModules PRs!

I just wanted to let you know that your changes (as of commit e12bacd) have been included in your existing downstream PRs.

modular-magician · 2018-10-18T22:29:08Z

I am a robot that works on MagicModules PRs!

I built this PR into one or more PRs on other repositories, and when those are closed, this PR will also be merged and closed.
depends: hashicorp/terraform-provider-google#2271
depends: hashicorp/terraform-provider-google-beta#12

modular-magician · 2018-10-19T00:32:15Z

I am (still) a robot that works on MagicModules PRs!

I just wanted to let you know that your changes (as of commit b0391d7) have been included in your existing downstream PRs.

danawillow · 2018-10-19T00:48:13Z

@chrisst, if you want to take another look, the downstream is looking a fair bit nicer now.

Merged PR #572.

Tracked submodules are build/puppet/_bundle build/puppet/auth build/puppet/bigquery build/puppet/compute build/puppet/sql build/puppet/storage build/puppet/spanner build/puppet/container build/puppet/dns build/puppet/pubsub build/puppet/resourcemanager build/chef/_bundle build/chef/auth build/chef/compute build/chef/sql build/chef/storage build/chef/spanner build/chef/container build/chef/dns build/chef/iam build/terraform-beta build/terraform build/ansible build/inspec.

add terraform-provider-google-beta to ci

6f36892

googlebot added the cla: yes label Sep 26, 2018

danawillow added 13 commits September 26, 2018 14:47

properly add beta submodule

3ccf867

add forgotten yml file

73048eb

remove bad submodule

f4c8a5c

actually add submodule

eca1e52

remove it again

0223d22

actually actually add submodule

ff18984

update submodule dir based on version

0439c11

quote version

f971c11

use other way of specifying submodule url

39fbaec

skip generation if product doesn't exist at specified version

248a4e3

generate tf at ga first, then at beta

1cc61a6

Merge branch 'master' into tfbeta

fff3560

copy git dir correctly

3da348f

modular-magician added the downstream-generated label Oct 2, 2018

maybe actually set the right version to generate, who really knows ba…

c0bbeb9

…sh anyway

danawillow added 3 commits October 2, 2018 20:17

try to run beta provider tests as part of terraform test job

0fde8ad

add beta provider in a few more places

80dc82f

make arrays correctly

ca035fb

fix issue with empty string becoming the word null

e178fe8

danawillow requested a review from nat-henderson October 3, 2018 17:22

nat-henderson reviewed Oct 3, 2018

View reviewed changes

danawillow added 3 commits October 3, 2018 13:56

more loops

3d616b9

add terraform-beta-generated output back

47aada0

Merge branch 'master' into tfbeta

8275eec

danawillow and others added 2 commits October 17, 2018 15:47

more indentation fixes

8c280d1

Add Storage ObjectAccessControl to Terraform (#562)

84e25bd

Merged PR #562.

danawillow requested review from rileykarson and chrisst October 17, 2018 23:13

fix expand for machine type in composer environment (#571)

62afbbb

Merged PR #571.

nat-henderson reviewed Oct 17, 2018

View reviewed changes

Terraform: Add storage, missing project resource test files to MM. (#582

19ace70

) Merged PR #582.

rileykarson approved these changes Oct 18, 2018

View reviewed changes

emilymye and others added 4 commits October 18, 2018 10:53

nested exclude for versioned properties (#575)

1f6c48c

Merged PR #575.

maps and variables in ci

abe3d0c

fix point-to-submodules

ff19553

fix create-pr

be6dd56

chrisst reviewed Oct 18, 2018

View reviewed changes

add missing -v

91291cc

small changes the magician tried to overwrite

234e48c

nat-henderson approved these changes Oct 19, 2018

View reviewed changes

rileykarson and others added 2 commits October 19, 2018 09:26

Add OiCS link generation to the Terraform docs. (#572)

b6e66c2

Merged PR #572.

Merge 234e48c into b6e66c2

1934858

chrisst approved these changes Oct 19, 2018

View reviewed changes

danawillow mentioned this pull request Oct 19, 2018

Terraform: More import formats and fix name validation for spanner databases #587

Closed

modular-magician added the automerged label Oct 19, 2018

modular-magician merged commit 3296816 into master Oct 19, 2018

danawillow deleted the tfbeta branch October 19, 2018 18:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add terraform-provider-google-beta to ci #502

add terraform-provider-google-beta to ci #502

danawillow commented Sep 26, 2018 •

edited

Loading

modular-magician commented Oct 2, 2018

modular-magician commented Oct 3, 2018

modular-magician commented Oct 3, 2018

danawillow commented Oct 3, 2018

nat-henderson Oct 3, 2018

nat-henderson Oct 3, 2018

nat-henderson Oct 3, 2018

nat-henderson Oct 3, 2018

modular-magician commented Oct 17, 2018

danawillow commented Oct 17, 2018

nat-henderson Oct 17, 2018

danawillow Oct 18, 2018

nat-henderson Oct 18, 2018

nat-henderson Oct 19, 2018

danawillow Oct 19, 2018

rileykarson left a comment

chrisst Oct 17, 2018

danawillow Oct 19, 2018

chrisst Oct 17, 2018

danawillow Oct 19, 2018

modular-magician commented Oct 18, 2018

modular-magician commented Oct 18, 2018 •

edited by danawillow

Loading

modular-magician commented Oct 19, 2018

danawillow commented Oct 19, 2018

add terraform-provider-google-beta to ci #502

add terraform-provider-google-beta to ci #502

Conversation

danawillow commented Sep 26, 2018 • edited Loading

[all]

[terraform]

[terraform-beta]

[puppet]

[puppet-bigquery]

[puppet-compute]

[puppet-container]

[puppet-dns]

[puppet-logging]

[puppet-pubsub]

[puppet-resourcemanager]

[puppet-sql]

[puppet-storage]

[chef]

[chef-compute]

[chef-container]

[chef-dns]

[chef-logging]

[chef-spanner]

[chef-sql]

[chef-storage]

[ansible]

modular-magician commented Oct 2, 2018

modular-magician commented Oct 3, 2018

modular-magician commented Oct 3, 2018

danawillow commented Oct 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

modular-magician commented Oct 17, 2018

danawillow commented Oct 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rileykarson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

modular-magician commented Oct 18, 2018

modular-magician commented Oct 18, 2018 • edited by danawillow Loading

modular-magician commented Oct 19, 2018

danawillow commented Oct 19, 2018

danawillow commented Sep 26, 2018 •

edited

Loading

modular-magician commented Oct 18, 2018 •

edited by danawillow

Loading