-
Notifications
You must be signed in to change notification settings - Fork 910
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
43 changed files
with
2,222 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| # Load testing an application running on an autopilot cluster | ||
|
|
||
| This blueprint creates an Autopilot cluster with Google-managed Prometheus enabled and install an application that scales as the traffic that is hitting the load balancer exposing it grows. It also installs the tooling required to distributed load test with [locust](https://locust.io) on that application and the monitoring tooling required to observe how things evolve in the cluster during the load test. Ansible is used to install the application and all the tooling on a management VM. | ||
|
|
||
| The diagram below depicts the architecture. | ||
|
|
||
|  | ||
|
|
||
| ## Running the blueprint | ||
|
|
||
| 1. Clone this repository or [open it in cloud shell](https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2Fterraform-google-modules%2Fcloud-foundation-fabric&cloudshell_print=cloud-shell-readme.txt&cloudshell_working_dir=blueprints%2Fgke%2Fautopilot), then go through the following steps to create resources: | ||
|
|
||
| 2. Initialize the terraform configuration | ||
|
|
||
| ``` | ||
| terraform init | ||
| ``` | ||
| 3. Apply the terraform configuration | ||
| ``` | ||
| terraform apply -var project_id=my-project-id | ||
| ``` | ||
| 4. Copy the IP addresses for grafana, the locust master. | ||
| 4. Change to the ansible directory and run the following command | ||
| ``` | ||
| ansible-playbook -v playbook.yaml | ||
| ``` | ||
| 5. Open to the locust master web interface url in your browser and start the load test | ||
| 6. SSH to the management VM | ||
| ``` | ||
| gcloud compute ssh mgmt --project my-project | ||
| ``` | ||
| 7. Run the following command to check that the application pods are running on different nodes than the load testing and monitoring tooling. | ||
| ``` | ||
| kubectl get pods -A -o wide | ||
| ``` | ||
| 8. Run the following command to see how the application pods scale | ||
| ``` | ||
| kubectl get hpa -n sample -w | ||
| ``` | ||
| 9. Run the following command to see how the cluster nodes scale | ||
| ``` | ||
| kubectl get nodes -n | ||
| ``` | ||
| Alternatively you can also check all the above using the dashboards available in grafana. | ||
| <!-- BEGIN TFDOC --> | ||
| ## Variables | ||
| | name | description | type | required | default | | ||
| |---|---|:---:|:---:|:---:| | ||
| | [project_id](variables.tf#L68) | Project ID. | <code>string</code> | ✓ | | | ||
| | [cluster_network_config](variables.tf#L17) | Cluster network configuration. | <code title="object({ nodes_cidr_block = string pods_cidr_block = string services_cidr_block = string master_authorized_cidr_blocks = map(string) master_cidr_block = string })">object({…})</code> | | <code title="{ nodes_cidr_block = "10.0.1.0/24" pods_cidr_block = "172.16.0.0/20" services_cidr_block = "192.168.0.0/24" master_authorized_cidr_blocks = { internal = "10.0.0.0/8" } master_cidr_block = "10.0.0.0/28" }">{…}</code> | | ||
| | [mgmt_server_config](variables.tf#L37) | Management server configuration. | <code title="object({ disk_size = number disk_type = string image = string instance_type = string })">object({…})</code> | | <code title="{ disk_size = 50 disk_type = "pd-ssd" image = "projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts" instance_type = "n1-standard-2" }">{…}</code> | | ||
| | [mgmt_subnet_cidr_block](variables.tf#L53) | Management subnet IP CIDR range. | <code>string</code> | | <code>"10.0.2.0/24"</code> | | ||
| | [project_create](variables.tf#L59) | Parameters for the creation of the new project. | <code title="object({ billing_account_id = string parent = string })">object({…})</code> | | <code>null</code> | | ||
| | [region](variables.tf#L73) | Region. | <code>string</code> | | <code>"europe-west1"</code> | | ||
| | [zone](variables.tf#L79) | Zone. | <code>string</code> | | <code>"europe-west1-c"</code> | | ||
| ## Outputs | ||
| | name | description | sensitive | | ||
| |---|---|:---:| | ||
| | [urls](outputs.tf#L17) | Grafanam, locust and application URLs. | | | ||
| <!-- END TFDOC --> | ||
| ## Test | ||
| ```hcl | ||
| module "test" { | ||
| source = "./fabric/blueprints/gke/autopilot" | ||
| project_create = { | ||
| billing_account_id = "12345-12345-12345" | ||
| parent = "folders/123456789" | ||
| } | ||
| project_id = "my-project" | ||
| } | ||
| # tftest modules=10 resources=30 | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| /** | ||
| * Copyright 2023 Google LLC | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| # tfdoc:file:description Ansible generated files. | ||
|
|
||
| resource "local_file" "vars_file" { | ||
| content = yamlencode({ | ||
| cluster = module.cluster.name | ||
| region = var.region | ||
| project_id = module.project.project_id | ||
| app_url = local.urls["app"] | ||
| }) | ||
| filename = "${path.module}/ansible/vars/vars.yaml" | ||
| file_permission = "0666" | ||
| } | ||
|
|
||
| resource "local_file" "gssh_file" { | ||
| content = templatefile("${path.module}/templates/gssh.sh.tpl", { | ||
| project_id = module.project.project_id | ||
| zone = var.zone | ||
| }) | ||
| filename = "${path.module}/ansible/gssh.sh" | ||
| file_permission = "0777" | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| [defaults] | ||
| inventory = inventory/hosts.ini | ||
| timeout = 900 | ||
|
|
||
| [ssh_connection] | ||
| pipelining = True | ||
| ssh_executable = ./gssh.sh | ||
| transfer_method = piped |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| mgmt |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,128 @@ | ||
| # Copyright 2022 Google LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| - hosts: mgmt | ||
| gather_facts: "no" | ||
| vars_files: | ||
| - vars/vars.yaml | ||
| environment: | ||
| USE_GKE_GCLOUD_AUTH_PLUGIN: True | ||
| tasks: | ||
| - name: Download the Google Cloud SDK package repository signing key | ||
| get_url: | ||
| url: https://packages.cloud.google.com/apt/doc/apt-key.gpg | ||
| dest: /usr/share/keyrings/cloud.google.gpg | ||
| force: yes | ||
| become: true | ||
| become_user: root | ||
| - name: Add Google Cloud SDK package repository source | ||
| apt_repository: | ||
| filename: google-cloud-sdk | ||
| repo: "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | ||
| state: present | ||
| update_cache: yes | ||
| become: true | ||
| become_user: root | ||
| - name: Install dependencies | ||
| apt: | ||
| pkg: | ||
| - google-cloud-sdk-gke-gcloud-auth-plugin | ||
| - kubectl | ||
| state: present | ||
| become: true | ||
| become_user: root | ||
| - name: Enable bash completion for kubectl | ||
| shell: | ||
| cmd: kubectl completion bash > /etc/bash_completion.d/kubectl | ||
| creates: /etc/bash_completion.d/kubectl | ||
| become: true | ||
| become_user: root | ||
| - name: Get cluster credentials | ||
| shell: > | ||
| gcloud container clusters get-credentials {{ cluster }} | ||
| --region {{ region }} | ||
| --project {{ project_id }} | ||
| --internal-ip | ||
| - name: Render templates | ||
| template: | ||
| src: ../bundle/{{ item }}/kustomization.yaml.j2 | ||
| dest: ../bundle/{{ item }}/kustomization.yaml | ||
| delegate_to: localhost | ||
| with_items: | ||
| - monitoring | ||
| - locust | ||
| - name: Remove bundle locally | ||
| local_action: | ||
| module: file | ||
| path: ../bundle.tar.gz | ||
| state: absent | ||
| - name: Archive bundle locally | ||
| archive: | ||
| path: ../bundle | ||
| dest: ../bundle.tar.gz | ||
| delegate_to: localhost | ||
| - name: Unarchive bundle remotely | ||
| unarchive: | ||
| src: ../bundle.tar.gz | ||
| dest: ~/ | ||
| - name: Build locust image | ||
| shell: > | ||
| gcloud builds submit --tag {{ region }}-docker.pkg.dev/{{ project_id }}/registry/load-test:latest \ | ||
| --project {{ project_id }} . | ||
| args: | ||
| chdir: ~/bundle/locust/image | ||
| - name: Enable scraping of kubelet and cAdvisor metrics | ||
| shell: > | ||
| kubectl patch operatorconfig config | ||
| -n gmp-public | ||
| --type=merge | ||
| -p '{"collection":{"kubeletScraping":{"interval": "30s"}}}' | ||
| - name: Deploy monitoring tooling | ||
| shell: > | ||
| kubectl apply -k . | ||
| args: | ||
| chdir: ~/bundle/monitoring | ||
| - name: Deploy app | ||
| shell: > | ||
| kubectl apply -k . | ||
| args: | ||
| chdir: ~/bundle/app | ||
| - name: Get forwarding rule name | ||
| shell: > | ||
| while true; do | ||
| forwarding_rule_name=$(kubectl get ingress -n sample -o=jsonpath='{.items[0].metadata.annotations.ingress\.kubernetes\.io\/forwarding-rule}') | ||
| if [ -n "$forwarding_rule_name" ]; then | ||
| echo $forwarding_rule_name | ||
| break | ||
| fi | ||
| sleep 10 | ||
| done | ||
| register: forwarding_rule_name_output | ||
| - name: Set fact forwarding_url_name | ||
| set_fact: | ||
| forwarding_rule_name: "{{ forwarding_rule_name_output.stdout }}" | ||
| - name: Render template (HPA) | ||
| template: | ||
| src: ../bundle/app/hpa.yaml.j2 | ||
| dest: ~/bundle/app/hpa.yaml | ||
| - name: Apply HPA manifest | ||
| shell: > | ||
| kubectl apply -f hpa.yaml | ||
| args: | ||
| chdir: ~/bundle/app | ||
| - name: Deploy locust | ||
| shell: > | ||
| kubectl apply -k . | ||
| args: | ||
| chdir: ~/bundle/locust |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| # Copyright 2023 Google LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| apiVersion: autoscaling/v2 | ||
| kind: HorizontalPodAutoscaler | ||
| metadata: | ||
| name: nginx | ||
| namespace: sample | ||
| spec: | ||
| scaleTargetRef: | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| name: nginx | ||
| minReplicas: 1 | ||
| maxReplicas: 50 | ||
| metrics: | ||
| - type: External | ||
| external: | ||
| metric: | ||
| name: loadbalancing.googleapis.com|https|request_count | ||
| selector: | ||
| matchLabels: | ||
| resource.labels.forwarding_rule_name: {{ forwarding_rule_name }} | ||
| target: | ||
| type: AverageValue | ||
| averageValue: 5 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| # Copyright 2023 Google LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| apiVersion: cloud.google.com/v1 | ||
| kind: BackendConfig | ||
| metadata: | ||
| name: backendconfig | ||
| namespace: sample | ||
| spec: | ||
| healthCheck: | ||
| requestPath: / | ||
| port: 80 | ||
| type: HTTP | ||
| logging: | ||
| enable: true | ||
| sampleRate: 0.5 | ||
| --- | ||
| apiVersion: networking.k8s.io/v1 | ||
| kind: Ingress | ||
| metadata: | ||
| annotations: | ||
| kubernetes.io/ingress.global-static-ip-name: "app" | ||
| kubernetes.io/ingress.allow-http: "true" | ||
| name: ingress | ||
| namespace: sample | ||
| spec: | ||
| defaultBackend: | ||
| service: | ||
| name: nginx | ||
| port: | ||
| name: web |
Oops, something went wrong.