Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding placement_policy for GKE nodepools (ex: GPU compact placement or TPU topology) #2405

Merged
merged 10 commits into from
Jul 3, 2024
Merged

Adding placement_policy for GKE nodepools (ex: GPU compact placement or TPU topology) #2405

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
d491b02
Adding placement policy to gke-nodepool module
legranda-adeo Jun 27, 2024
71022c7
Adding placement policy for GKE nodepool
legranda-adeo Jul 2, 2024
03e261a
updated README
legranda-adeo Jul 2, 2024
584e3eb
variables for placement_policy
legranda-adeo Jul 2, 2024
80e3be3
formatting
legranda-adeo Jul 2, 2024
f2297c1
Updated README
aurelienlegrand Jul 2, 2024
dd48bf7
fixing typo
aurelienlegrand Jul 2, 2024
a0b5839
Merge branch 'master' into gke-nodepools-tpu
aurelienlegrand Jul 2, 2024
bd540a3
removing useless trys
aurelienlegrand Jul 3, 2024
c384769
Merge branch 'gke-nodepools-tpu' of https://github.com/GoogleCloudPla…
aurelienlegrand Jul 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions modules/gke-nodepool/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ module "cluster-1-nodepool-gpu-1" {
|---|---|:---:|:---:|:---:|
| [cluster_name](variables.tf#L23) | Cluster name. | <code>string</code> | ✓ | |
| [location](variables.tf#L48) | Cluster location. | <code>string</code> | ✓ | |
| [project_id](variables.tf#L177) | Cluster project id. | <code>string</code> | ✓ | |
| [project_id](variables.tf#L181) | Cluster project id. | <code>string</code> | ✓ | |
| [cluster_id](variables.tf#L17) | Cluster id. Optional, but providing cluster_id is recommended to prevent cluster misconfiguration in some of the edge cases. | <code>string</code> | | <code>null</code> |
| [gke_version](variables.tf#L28) | Kubernetes nodes version. Ignored if auto_upgrade is set in management_config. | <code>string</code> | | <code>null</code> |
| [k8s_labels](variables.tf#L34) | Kubernetes labels applied to each node. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> |
Expand All @@ -153,13 +153,13 @@ module "cluster-1-nodepool-gpu-1" {
| [node_config](variables.tf#L65) | Node-level configuration. | <code title="object&#40;&#123;&#10; boot_disk_kms_key &#61; optional&#40;string&#41;&#10; disk_size_gb &#61; optional&#40;number&#41;&#10; disk_type &#61; optional&#40;string&#41;&#10; ephemeral_ssd_count &#61; optional&#40;number&#41;&#10; gcfs &#61; optional&#40;bool, false&#41;&#10; guest_accelerator &#61; optional&#40;object&#40;&#123;&#10; count &#61; number&#10; type &#61; string&#10; gpu_driver &#61; optional&#40;object&#40;&#123;&#10; version &#61; string&#10; partition_size &#61; optional&#40;string&#41;&#10; max_shared_clients_per_gpu &#61; optional&#40;number&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10; local_nvme_ssd_count &#61; optional&#40;number&#41;&#10; gvnic &#61; optional&#40;bool, false&#41;&#10; image_type &#61; optional&#40;string&#41;&#10; kubelet_config &#61; optional&#40;object&#40;&#123;&#10; cpu_manager_policy &#61; string&#10; cpu_cfs_quota &#61; optional&#40;bool&#41;&#10; cpu_cfs_quota_period &#61; optional&#40;string&#41;&#10; pod_pids_limit &#61; optional&#40;number&#41;&#10; &#125;&#41;&#41;&#10; linux_node_config &#61; optional&#40;object&#40;&#123;&#10; sysctls &#61; optional&#40;map&#40;string&#41;&#41;&#10; cgroup_mode &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10; local_ssd_count &#61; optional&#40;number&#41;&#10; machine_type &#61; optional&#40;string&#41;&#10; metadata &#61; optional&#40;map&#40;string&#41;&#41;&#10; min_cpu_platform &#61; optional&#40;string&#41;&#10; preemptible &#61; optional&#40;bool&#41;&#10; sandbox_config_gvisor &#61; optional&#40;bool&#41;&#10; shielded_instance_config &#61; optional&#40;object&#40;&#123;&#10; enable_integrity_monitoring &#61; optional&#40;bool&#41;&#10; enable_secure_boot &#61; optional&#40;bool&#41;&#10; &#125;&#41;&#41;&#10; spot &#61; optional&#40;bool&#41;&#10; workload_metadata_config_mode &#61; optional&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; disk_type &#61; &#34;pd-balanced&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
| [node_count](variables.tf#L124) | Number of nodes per instance group. Initial value can only be changed by recreation, current is ignored when autoscaling is used. | <code title="object&#40;&#123;&#10; current &#61; optional&#40;number&#41;&#10; initial &#61; number&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; initial &#61; 1&#10;&#125;">&#123;&#8230;&#125;</code> |
| [node_locations](variables.tf#L136) | Node locations. | <code>list&#40;string&#41;</code> | | <code>null</code> |
| [nodepool_config](variables.tf#L142) | Nodepool-level configuration. | <code title="object&#40;&#123;&#10; autoscaling &#61; optional&#40;object&#40;&#123;&#10; location_policy &#61; optional&#40;string&#41;&#10; max_node_count &#61; optional&#40;number&#41;&#10; min_node_count &#61; optional&#40;number&#41;&#10; use_total_nodes &#61; optional&#40;bool, false&#41;&#10; &#125;&#41;&#41;&#10; management &#61; optional&#40;object&#40;&#123;&#10; auto_repair &#61; optional&#40;bool&#41;&#10; auto_upgrade &#61; optional&#40;bool&#41;&#10; &#125;&#41;&#41;&#10; upgrade_settings &#61; optional&#40;object&#40;&#123;&#10; max_surge &#61; number&#10; max_unavailable &#61; number&#10; &#125;&#41;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [pod_range](variables.tf#L164) | Pod secondary range configuration. | <code title="object&#40;&#123;&#10; secondary_pod_range &#61; object&#40;&#123;&#10; name &#61; string&#10; cidr &#61; optional&#40;string&#41;&#10; create &#61; optional&#40;bool&#41;&#10; enable_private_nodes &#61; optional&#40;bool&#41;&#10; &#125;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [reservation_affinity](variables.tf#L182) | Configuration of the desired reservation which instances could take capacity from. | <code title="object&#40;&#123;&#10; consume_reservation_type &#61; string&#10; key &#61; optional&#40;string&#41;&#10; values &#61; optional&#40;list&#40;string&#41;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [service_account](variables.tf#L192) | Nodepool service account. If this variable is set to null, the default GCE service account will be used. If set and email is null, a service account will be created. If scopes are null a default will be used. | <code title="object&#40;&#123;&#10; create &#61; optional&#40;bool, false&#41;&#10; email &#61; optional&#40;string&#41;&#10; oauth_scopes &#61; optional&#40;list&#40;string&#41;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>&#123;&#125;</code> |
| [sole_tenant_nodegroup](variables.tf#L203) | Sole tenant node group. | <code>string</code> | | <code>null</code> |
| [tags](variables.tf#L209) | Network tags applied to nodes. | <code>list&#40;string&#41;</code> | | <code>null</code> |
| [taints](variables.tf#L215) | Kubernetes taints applied to all nodes. | <code title="map&#40;object&#40;&#123;&#10; value &#61; string&#10; effect &#61; string&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [nodepool_config](variables.tf#L142) | Nodepool-level configuration. | <code title="object&#40;&#123;&#10; autoscaling &#61; optional&#40;object&#40;&#123;&#10; location_policy &#61; optional&#40;string&#41;&#10; max_node_count &#61; optional&#40;number&#41;&#10; min_node_count &#61; optional&#40;number&#41;&#10; use_total_nodes &#61; optional&#40;bool, false&#41;&#10; &#125;&#41;&#41;&#10; management &#61; optional&#40;object&#40;&#123;&#10; auto_repair &#61; optional&#40;bool&#41;&#10; auto_upgrade &#61; optional&#40;bool&#41;&#10; &#125;&#41;&#41;&#10; placement_policy &#61; optional&#40;object&#40;&#123;&#10; type &#61; string&#10; policy_name &#61; optional&#40;string&#41;&#10; tpu_topology &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10; upgrade_settings &#61; optional&#40;object&#40;&#123;&#10; max_surge &#61; number&#10; max_unavailable &#61; number&#10; &#125;&#41;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [pod_range](variables.tf#L168) | Pod secondary range configuration. | <code title="object&#40;&#123;&#10; secondary_pod_range &#61; object&#40;&#123;&#10; name &#61; string&#10; cidr &#61; optional&#40;string&#41;&#10; create &#61; optional&#40;bool&#41;&#10; enable_private_nodes &#61; optional&#40;bool&#41;&#10; &#125;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [reservation_affinity](variables.tf#L186) | Configuration of the desired reservation which instances could take capacity from. | <code title="object&#40;&#123;&#10; consume_reservation_type &#61; string&#10; key &#61; optional&#40;string&#41;&#10; values &#61; optional&#40;list&#40;string&#41;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [service_account](variables.tf#L196) | Nodepool service account. If this variable is set to null, the default GCE service account will be used. If set and email is null, a service account will be created. If scopes are null a default will be used. | <code title="object&#40;&#123;&#10; create &#61; optional&#40;bool, false&#41;&#10; email &#61; optional&#40;string&#41;&#10; oauth_scopes &#61; optional&#40;list&#40;string&#41;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>&#123;&#125;</code> |
| [sole_tenant_nodegroup](variables.tf#L207) | Sole tenant node group. | <code>string</code> | | <code>null</code> |
| [tags](variables.tf#L213) | Network tags applied to nodes. | <code>list&#40;string&#41;</code> | | <code>null</code> |
| [taints](variables.tf#L219) | Kubernetes taints applied to all nodes. | <code title="map&#40;object&#40;&#123;&#10; value &#61; string&#10; effect &#61; string&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |

## Outputs

Expand Down
10 changes: 9 additions & 1 deletion modules/gke-nodepool/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,6 @@ resource "google_container_node_pool" "nodepool" {
initial_node_count = var.node_count.initial
node_count = var.node_count.current
node_locations = var.node_locations
# placement_policy = var.nodepool_config.placement_policy

dynamic "autoscaling" {
for_each = (
Expand Down Expand Up @@ -129,6 +128,15 @@ resource "google_container_node_pool" "nodepool" {
}
}

dynamic "placement_policy" {
for_each = try(var.nodepool_config.placement_policy, null) != null ? [""] : []
content {
type = try(var.nodepool_config.placement_policy.type, null)
policy_name = try(var.nodepool_config.placement_policy.policy_name, null)
tpu_topology = try(var.nodepool_config.placement_policy.tpu_topology, null)
wiktorn marked this conversation as resolved.
Show resolved Hide resolved
}
}

node_config {
boot_disk_kms_key = var.node_config.boot_disk_kms_key
disk_size_gb = var.node_config.disk_size_gb
Expand Down
6 changes: 5 additions & 1 deletion modules/gke-nodepool/variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,11 @@ variable "nodepool_config" {
auto_repair = optional(bool)
auto_upgrade = optional(bool)
}))
# placement_policy = optional(bool)
placement_policy = optional(object({
type = string
policy_name = optional(string)
tpu_topology = optional(string)
}))
upgrade_settings = optional(object({
max_surge = number
max_unavailable = number
Expand Down