Added autopilot blueprint

.gitignore

@@ -49,3 +49,8 @@ blueprints/apigee/hybrid-gke/apiproxy.zip
 blueprints/apigee/hybrid-gke/deploy-apiproxy.sh
 blueprints/apigee/hybrid-gke/ansible/gssh.sh
 blueprints/apigee/hybrid-gke/ansible/vars/vars.yaml
+blueprints/gke/autopilot/ansible/gssh.sh
+blueprints/gke/autopilot/ansible/vars/vars.yaml
+blueprints/gke/autopilot/bundle/monitoring/kustomization.yaml
+blueprints/gke/autopilot/bundle/locust/kustomization.yaml
+blueprints/gke/autopilot/bundle.tar.gz

blueprints/gke/autopilot/README.md

@@ -0,0 +1,94 @@
+# Load testing an application running on an autopilot cluster
+
+This blueprint creates an Autopilot cluster with Google-managed Prometheus enabled and install an application that scales as the traffic that is hitting the load balancer exposing it grows. It also installs the tooling required to distributed load test with [locust](https://locust.io) on that application and the monitoring tooling required to observe how things evolve in the cluster during the load test. Ansible is used to install the application and all the tooling on a management VM. 
+
+The diagram below depicts the architecture.
+
+![Diagram](./diagram.png)
+
+## Running the blueprint
+
+1. Clone this repository or [open it in cloud shell](https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2Fterraform-google-modules%2Fcloud-foundation-fabric&cloudshell_print=cloud-shell-readme.txt&cloudshell_working_dir=blueprints%2Fgke%2Fautopilot), then go through the following steps to create resources:
+
+2. Initialize the terraform configuration
+
+    ```
+    terraform init
+    ```
+
+3. Apply the terraform configuration
+
+    ```
+    terraform apply -var project_id=my-project-id
+    ```
+
+4. Copy the IP addresses for grafana, the locust master.    
+
+4. Change to the ansible directory and run the following command
+
+    ```
+    ansible-playbook -v playbook.yaml
+    ```
+
+5. Open to the locust master web interface url in your browser and start the load test
+
+
+6. SSH to the management VM
+
+        ```
+        gcloud compute ssh mgmt --project my-project
+        ```
+
+7. Run the following command to check that the application pods are running on different nodes than the load testing and monitoring tooling.
+
+        ```
+        kubectl get pods -A -o wide
+        ```    
+  
+8. Run the following command to see how the application pods scale
+
+        ```
+        kubectl get hpa -n sample -w
+        ```
+
+9. Run the following command to see how the cluster nodes scale
+
+        ```
+        kubectl get nodes -n
+        ```
+
+Alternatively you can also check all the above using the dashboards available in grafana.
+<!-- BEGIN TFDOC -->
+
+## Variables
+
+| name | description | type | required | default |
+|---|---|:---:|:---:|:---:|
+| [project_id](variables.tf#L68) | Project ID. | <code>string</code> | ✓ |  |
+| [cluster_network_config](variables.tf#L17) | Cluster network configuration. | <code title="object&#40;&#123;&#10;  nodes_cidr_block              &#61; string&#10;  pods_cidr_block               &#61; string&#10;  services_cidr_block           &#61; string&#10;  master_authorized_cidr_blocks &#61; map&#40;string&#41;&#10;  master_cidr_block             &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code title="&#123;&#10;  nodes_cidr_block    &#61; &#34;10.0.1.0&#47;24&#34;&#10;  pods_cidr_block     &#61; &#34;172.16.0.0&#47;20&#34;&#10;  services_cidr_block &#61; &#34;192.168.0.0&#47;24&#34;&#10;  master_authorized_cidr_blocks &#61; &#123;&#10;    internal &#61; &#34;10.0.0.0&#47;8&#34;&#10;  &#125;&#10;  master_cidr_block &#61; &#34;10.0.0.0&#47;28&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
+| [mgmt_server_config](variables.tf#L37) | Management server configuration. | <code title="object&#40;&#123;&#10;  disk_size     &#61; number&#10;  disk_type     &#61; string&#10;  image         &#61; string&#10;  instance_type &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code title="&#123;&#10;  disk_size     &#61; 50&#10;  disk_type     &#61; &#34;pd-ssd&#34;&#10;  image         &#61; &#34;projects&#47;ubuntu-os-cloud&#47;global&#47;images&#47;family&#47;ubuntu-2204-lts&#34;&#10;  instance_type &#61; &#34;n1-standard-2&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
+| [mgmt_subnet_cidr_block](variables.tf#L53) | Management subnet IP CIDR range. | <code>string</code> |  | <code>&#34;10.0.2.0&#47;24&#34;</code> |
+| [project_create](variables.tf#L59) | Parameters for the creation of the new project. | <code title="object&#40;&#123;&#10;  billing_account_id &#61; string&#10;  parent             &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code>null</code> |
+| [region](variables.tf#L73) | Region. | <code>string</code> |  | <code>&#34;europe-west1&#34;</code> |
+| [zone](variables.tf#L79) | Zone. | <code>string</code> |  | <code>&#34;europe-west1-c&#34;</code> |
+
+## Outputs
+
+| name | description | sensitive |
+|---|---|:---:|
+| [urls](outputs.tf#L17) |  |  |
+
+<!-- END TFDOC -->
+## Test
+
+```hcl
+module "test" {
+  source = "./fabric/blueprints/gke/autopilot"
+  project_create = {
+    billing_account_id = "12345-12345-12345"
+    parent             = "folders/123456789"
+  }
+  project_id = "my-project"
+}
+# tftest modules=10 resources=30
+```

blueprints/gke/autopilot/ansible.tf

@@ -0,0 +1,37 @@
+/**
+ * Copyright 2023 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+# tfdoc:file:description Ansible generated files.
+
+resource "local_file" "vars_file" {
+  content = yamlencode({
+    cluster    = module.cluster.name
+    region     = var.region
+    project_id = module.project.project_id
+    app_url   = local.urls["app"]
+  })
+  filename        = "${path.module}/ansible/vars/vars.yaml"
+  file_permission = "0666"
+}
+
+resource "local_file" "gssh_file" {
+  content = templatefile("${path.module}/templates/gssh.sh.tpl", {
+    project_id = module.project.project_id
+    zone       = var.zone
+  })
+  filename        = "${path.module}/ansible/gssh.sh"
+  file_permission = "0777"
+}

blueprints/gke/autopilot/ansible/ansible.cfg

@@ -0,0 +1,8 @@
+[defaults]
+inventory = inventory/hosts.ini
+timeout = 900
+
+[ssh_connection]
+pipelining = True
+ssh_executable = ./gssh.sh
+transfer_method = piped

blueprints/gke/autopilot/ansible/inventory/hosts.ini

@@ -0,0 +1 @@
+mgmt

blueprints/gke/autopilot/ansible/playbook.yaml

@@ -0,0 +1,128 @@
+# Copyright 2022 Google LLC
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- hosts: mgmt
+  gather_facts: "no"
+  vars_files:
+    - vars/vars.yaml
+  environment:
+    USE_GKE_GCLOUD_AUTH_PLUGIN: True
+  tasks:
+    - name: Download the Google Cloud SDK package repository signing key
+      get_url:
+        url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
+        dest: /usr/share/keyrings/cloud.google.gpg
+        force: yes
+      become: true
+      become_user: root
+    - name: Add Google Cloud SDK package repository source
+      apt_repository:
+        filename: google-cloud-sdk
+        repo: "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main"
+        state: present
+        update_cache: yes
+      become: true
+      become_user: root  
+    - name: Install dependencies
+      apt:
+        pkg:
+          - google-cloud-sdk-gke-gcloud-auth-plugin
+          - kubectl
+        state: present
+      become: true
+      become_user: root
+    - name: Enable bash completion for kubectl
+      shell:
+        cmd: kubectl completion bash > /etc/bash_completion.d/kubectl
+        creates: /etc/bash_completion.d/kubectl      
+      become: true
+      become_user: root        
+    - name: Get cluster credentials
+      shell: >
+        gcloud container clusters get-credentials {{ cluster }} 
+        --region {{ region }} 
+        --project {{ project_id }} 
+        --internal-ip      
+    - name: Render templates
+      template:
+        src: ../bundle/{{ item }}/kustomization.yaml.j2
+        dest: ../bundle/{{ item }}/kustomization.yaml
+      delegate_to: localhost
+      with_items:
+        - monitoring
+        - locust
+    - name: Remove bundle locally
+      local_action: 
+        module: file 
+        path: ../bundle.tar.gz 
+        state: absent
+    - name: Archive bundle locally
+      archive:
+        path: ../bundle
+        dest: ../bundle.tar.gz
+      delegate_to: localhost
+    - name: Unarchive bundle remotely
+      unarchive:
+        src: ../bundle.tar.gz
+        dest: ~/  
+    - name: Build locust image
+      shell: >
+        gcloud builds submit --tag {{ region }}-docker.pkg.dev/{{ project_id }}/registry/load-test:latest \ 
+        --project {{ project_id }} .
+      args:
+        chdir: ~/bundle/locust/image
+    - name: Enable scraping of kubelet and cAdvisor metrics
+      shell: >
+        kubectl patch operatorconfig config
+        -n gmp-public
+        --type=merge
+        -p '{"collection":{"kubeletScraping":{"interval": "30s"}}}'
+    - name: Deploy monitoring tooling
+      shell: >
+        kubectl apply -k .
+      args:
+        chdir: ~/bundle/monitoring
+    - name: Deploy app
+      shell: >
+        kubectl apply -k .
+      args:
+        chdir: ~/bundle/app
+    - name: Get forwarding rule name
+      shell: >
+        while true; do
+            forwarding_rule_name=$(kubectl get ingress -n sample -o=jsonpath='{.items[0].metadata.annotations.ingress\.kubernetes\.io\/forwarding-rule}')
+            if [ -n "$forwarding_rule_name" ]; then
+              echo $forwarding_rule_name
+              break
+            fi
+            sleep 10
+        done        
+      register: forwarding_rule_name_output
+    - name: Set fact forwarding_url_name
+      set_fact:
+        forwarding_rule_name: "{{ forwarding_rule_name_output.stdout }}"
+    - name: Render template (HPA)
+      template:
+        src: ../bundle/app/hpa.yaml.j2
+        dest: ~/bundle/app/hpa.yaml
+    - name: Apply HPA manifest
+      shell: >
+        kubectl apply -f hpa.yaml
+      args:
+        chdir: ~/bundle/app      
+    - name: Deploy locust 
+      shell: >
+        kubectl apply -k .
+      args:
+        chdir: ~/bundle/locust      

blueprints/gke/autopilot/bundle/app/hpa.yaml.j2

@@ -0,0 +1,37 @@
+# Copyright 2023 Google LLC
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: nginx
+  namespace: sample
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: nginx
+  minReplicas: 1
+  maxReplicas: 50
+  metrics:
+  - type: External 
+    external:
+      metric: 
+        name: loadbalancing.googleapis.com|https|request_count
+        selector:
+          matchLabels:
+            resource.labels.forwarding_rule_name: {{ forwarding_rule_name }}
+      target:
+        type: AverageValue
+        averageValue: 5

blueprints/gke/autopilot/bundle/app/ingress.yaml

@@ -0,0 +1,42 @@
+# Copyright 2023 Google LLC
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: cloud.google.com/v1
+kind: BackendConfig
+metadata:
+  name: backendconfig
+  namespace: sample
+spec:
+  healthCheck:
+    requestPath: /
+    port: 80
+    type: HTTP
+  logging:
+    enable: true
+    sampleRate: 0.5   
+---      
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  annotations:
+    kubernetes.io/ingress.global-static-ip-name: "app"
+    kubernetes.io/ingress.allow-http: "true"
+  name: ingress
+  namespace: sample
+spec:
+  defaultBackend:
+    service:
+      name: nginx
+      port:
+        name: web

blueprints/gke/autopilot/bundle/app/kustomization.yaml

@@ -0,0 +1,18 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https:#www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+resources:
+  - namespace.yaml
+  - nginx.yaml
+  - ingress.yaml