Initial replacement for CI/CD stage

fast/extras/00-cicd-github/README.md

@@ -0,0 +1,92 @@
+# FAST GitHub repository management
+
+This small extra stage allows creation and management of GitHub repositories used to host FAST stage code, including initial population of files and rewriting of module sources.
+
+This stage is designed for quick repository creation in a GitHub organization, and is not suited for medium or long-term repository management especially if you enable initial population of files.
+
+## Initial population caveats
+
+Initial file population of repositories is controlled via the `populate_from` attribute, and needs a bit of care:
+
+- never run this stage gain with the same variables used for population once the repository starts being used,  as **Terraform will manage file state and revert any changes at each apply**, which is probably not what you want.
+- be mindful when enabling initial population of the modules repository, as the number of resulting files to manage is very close to the GitHub hourly limit for their API
+
+The scenario for which this stage has been designed is one-shot creation and/or population of stage repositories, running it multiple times with different variables and Terraform states if incremental creation is needed for subsequent FAST stages (e.g. GKE, data platform, etc.).
+
+## GitHub provider credentials
+
+A [GitHub token](https://github.com/settings/tokens) is needed to authenticate against their API. The token needs organization-level permissions, like shown in this screenshot:
+
+<p align="center">
+  <img src="github_token.png" alt="GitHub token scopes.">
+</p>
+
+## Variable configuration
+
+The `organization` required variable sets the GitHub organization where repositories will be created, and is used to configure the Terraform provider.
+
+The `repositories` variable is where you configure which repositories to create, whether initial population of files is desired, and which repository is used to host modules.
+
+This is an example that creates repositories for stages 00 and 01, defines an existing repositories as the source for modules, and populates initial files for stages 00, 01, and 02:
+
+```hcl
+organization = "ludomagno"
+repositories = {
+  fast_00_bootstrap = {
+    create_options = {
+      description = "FAST bootstrap."
+      features = {
+        issues = true
+      }
+    }
+    populate_from = "../../stages/00-bootstrap"
+  }
+  fast_01_resman = {
+    create_options = {
+      description = "FAST resource management."
+      features = {
+        issues = true
+      }
+    }
+    populate_from = "../../stages/01-resman"
+  }
+  fast_02_networking = {
+    populate_from = "../../stages/02-networking-peering"
+  }
+  fast_modules = {
+    has_modules = true
+  }
+}
+```
+
+The `create_options` repository attribute controls creation: if the attribute is not present, the repository is assumed to be already existing.
+
+Initial population depends on a modules repository being configured, identified by the `has_modules` attribute, and on `populate_from` attributes in each repository where population is required, pointing to the folder holding the files to be committed.
+
+Finally, a `commit_config` variable is optional: it can be used to configure author, email and message used in commits for initial population of files, its defaults are probably fine for most use cases.
+
+## Modules secret
+
+When initial population is configured for a repository, this stage also adds a secret with the private key used to authenticate against the modules repository. This matches the configuration of the GitHub workflow files created for each FAST stage when CI/CD is enabled.
+
+<!-- TFDOC OPTS files:1 -->
+<!-- BEGIN TFDOC -->
+
+## Files
+
+| name | description | resources |
+|---|---|---|
+| [cicd-versions.tf](./cicd-versions.tf) | Provider version. |  |
+| [main.tf](./main.tf) | Module-level locals and resources. | <code>github_actions_secret</code> · <code>github_repository</code> · <code>github_repository_file</code> · <code>tls_private_key</code> |
+| [providers.tf](./providers.tf) | Provider configuration. |  |
+| [variables.tf](./variables.tf) | Module variables. |  |
+
+## Variables
+
+| name | description | type | required | default |
+|---|---|:---:|:---:|:---:|
+| [organization](variables.tf#L28) | GitHub organization. | <code>string</code> | ✓ |  |
+| [commmit_config](variables.tf#L17) | Configure commit metadata. | <code title="object&#40;&#123;&#10;  author  &#61; optional&#40;string, &#34;FAST loader&#34;&#41;&#10;  email   &#61; optional&#40;string, &#34;fast-loader&#64;fast.gcp.tf&#34;&#41;&#10;  message &#61; optional&#40;string, &#34;FAST initial loading&#34;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code>&#123;&#125;</code> |
+| [repositories](variables.tf#L33) | Repositories to create. | <code title="map&#40;object&#40;&#123;&#10;  create_options &#61; optional&#40;object&#40;&#123;&#10;    allow &#61; optional&#40;object&#40;&#123;&#10;      auto_merge   &#61; optional&#40;bool&#41;&#10;      merge_commit &#61; optional&#40;bool&#41;&#10;      rebase_merge &#61; optional&#40;bool&#41;&#10;      squash_merge &#61; optional&#40;bool&#41;&#10;    &#125;&#41;&#41;&#10;    auto_init   &#61; optional&#40;bool&#41;&#10;    description &#61; optional&#40;string&#41;&#10;    features &#61; optional&#40;object&#40;&#123;&#10;      issues   &#61; optional&#40;bool&#41;&#10;      projects &#61; optional&#40;bool&#41;&#10;      wiki     &#61; optional&#40;bool&#41;&#10;    &#125;&#41;&#41;&#10;    templates &#61; optional&#40;object&#40;&#123;&#10;      gitignore &#61; optional&#40;string, &#34;Terraform&#34;&#41;&#10;      license   &#61; optional&#40;string&#41;&#10;      repository &#61; optional&#40;object&#40;&#123;&#10;        name  &#61; string&#10;        owner &#61; string&#10;      &#125;&#41;&#41;&#10;    &#125;&#41;, &#123;&#125;&#41;&#10;    visibility &#61; optional&#40;string, &#34;private&#34;&#41;&#10;  &#125;&#41;&#41;&#10;  has_modules   &#61; optional&#40;bool, false&#41;&#10;  populate_from &#61; optional&#40;string&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> |  | <code>&#123;&#125;</code> |
+
+<!-- END TFDOC -->

fast/stages/00-cicd/outputs-gcs.tf → fast/extras/00-cicd-github/cicd-versions.tf

@@ -14,10 +14,16 @@
  * limitations under the License.
  */
 
-# tfdoc:file:description Output files persistence to automation GCS bucket.
+# tfdoc:file:description Provider version.
 
-resource "google_storage_bucket_object" "tfvars" {
-  bucket  = var.automation.outputs_bucket
-  name    = "tfvars/00-bootstrap.auto.tfvars.json"
-  content = jsonencode(local.tfvars)
+terraform {
+  required_version = ">= 1.3.1"
+  required_providers {
+    github = {
+      source  = "integrations/github"
+      version = "~> 4.0"
+    }
+  }
 }
+
+

fast/extras/00-cicd-github/main.tf

@@ -0,0 +1,124 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+locals {
+  _modules_repository = [
+    for k, v in var.repositories : local.repositories[k] if v.has_modules
+  ]
+  _repository_files = flatten([
+    for k, v in var.repositories : [
+      for f in concat(
+        [for f in fileset(path.module, "${v.populate_from}/*.md") : f],
+        [for f in fileset(path.module, "${v.populate_from}/*.tf") : f]
+        ) : {
+        repository = k
+        file       = f
+        name       = replace(f, "${v.populate_from}/", "")
+      }
+    ] if v.populate_from != null
+  ])
+  modules_repository = (
+    length(local._modules_repository) > 0
+    ? local._modules_repository.0
+    : null
+  )
+  repositories = {
+    for k, v in var.repositories :
+    k => v.create_options == null ? k : github_repository.default[k].name
+  }
+  repository_files = {
+    for k in local._repository_files :
+    "${k.repository}/${k.name}" => k
+    if !endswith(k.name, ".tf") || (
+      !startswith(k.name, "0") && k.name != "globals.tf"
+    )
+  }
+}
+
+resource "github_repository" "default" {
+  for_each = {
+    for k, v in var.repositories : k => v if v.create_options != null
+  }
+  name = each.key
+  description = (
+    each.value.create_options.description != null
+    ? each.value.create_options.description
+    : "FAST stage ${each.key}."
+  )
+  visibility         = each.value.create_options.visibility
+  auto_init          = each.value.create_options.auto_init
+  allow_auto_merge   = try(each.value.create_options.allow.auto_merge, null)
+  allow_merge_commit = try(each.value.create_options.allow.merge_commit, null)
+  allow_rebase_merge = try(each.value.create_options.allow.rebase_merge, null)
+  allow_squash_merge = try(each.value.create_options.allow.squash_merge, null)
+  has_issues         = try(each.value.create_options.features.issues, null)
+  has_projects       = try(each.value.create_options.features.projects, null)
+  has_wiki           = try(each.value.create_options.features.wiki, null)
+  gitignore_template = try(each.value.create_options.templates.gitignore, null)
+  license_template   = try(each.value.create_options.templates.license, null)
+
+  dynamic "template" {
+    for_each = (
+      try(each.value.create_options.templates.repository, null) != null
+      ? [""]
+      : []
+    )
+    content {
+      owner      = each.value.create_options.templates.repository.owner
+      repository = each.value.create_options.templates.repository.name
+    }
+  }
+}
+
+resource "tls_private_key" "default" {
+  count     = local.modules_repository != null ? 1 : 0
+  algorithm = "ED25519"
+}
+
+resource "github_actions_secret" "default" {
+  for_each = local.modules_repository == null ? {} : {
+    for k, v in local.repositories :
+    k => v if(
+      k != local.modules_repository &&
+      var.repositories[k].populate_from != null
+    )
+  }
+  repository      = local.repositories[local.modules_repository]
+  secret_name     = "CICD_MODULES_KEY"
+  plaintext_value = tls_private_key.default.0.private_key_openssh
+}
+
+resource "github_repository_file" "default" {
+  for_each = (
+    local.modules_repository == null ? {} : local.repository_files
+  )
+  repository = local.repositories[each.value.repository]
+  branch     = "main"
+  file       = each.value.name
+  content = (
+    endswith(each.value.name, ".tf") && local.modules_repository != null
+    ? replace(
+      file(each.value.file),
+      "/source\\s*=\\s*\"../../../",
+      "source = \"[email protected]:${var.organization}/${local.modules_repository}.git/"
+    )
+    : file(each.value.file)
+  )
+  commit_message      = "${var.commmit_config.message} (${each.value.name})"
+  commit_author       = var.commmit_config.author
+  commit_email        = var.commmit_config.email
+  overwrite_on_create = true
+}

fast/stages/00-cicd/main.tf → fast/extras/00-cicd-github/providers.tf

@@ -14,3 +14,8 @@
  * limitations under the License.
  */
 
+# tfdoc:file:description Provider configuration.
+
+provider "github" {
+  owner = var.organization
+}

fast/extras/00-cicd-github/variables.tf

@@ -0,0 +1,72 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+variable "commmit_config" {
+  description = "Configure commit metadata."
+  type = object({
+    author  = optional(string, "FAST loader")
+    email   = optional(string, "[email protected]")
+    message = optional(string, "FAST initial loading")
+  })
+  default  = {}
+  nullable = false
+}
+
+variable "organization" {
+  description = "GitHub organization."
+  type        = string
+}
+
+variable "repositories" {
+  description = "Repositories to create."
+  type = map(object({
+    create_options = optional(object({
+      allow = optional(object({
+        auto_merge   = optional(bool)
+        merge_commit = optional(bool)
+        rebase_merge = optional(bool)
+        squash_merge = optional(bool)
+      }))
+      auto_init   = optional(bool)
+      description = optional(string)
+      features = optional(object({
+        issues   = optional(bool)
+        projects = optional(bool)
+        wiki     = optional(bool)
+      }))
+      templates = optional(object({
+        gitignore = optional(string, "Terraform")
+        license   = optional(string)
+        repository = optional(object({
+          name  = string
+          owner = string
+        }))
+      }), {})
+      visibility = optional(string, "private")
+    }))
+    has_modules   = optional(bool, false)
+    populate_from = optional(string)
+  }))
+  default  = {}
+  nullable = true
+  validation {
+    condition = alltrue([
+      for k, v in var.repositories :
+      try(regex("^[a-zA-Z0-9_.]+$", k), null) != null
+    ])
+    error_message = "Repository names must match '^[a-zA-Z0-9_.]+$'."
+  }
+}

fast/extras/README.md

@@ -0,0 +1,5 @@
+# FAST extra stages
+
+This folder contains additional helper stages for FAST, which can be used to simplify specific operational tasks:
+
+- [GitHub repository management](./00-cicd-github/)