[feature request] Access to google_dataproc_cluster master/worker node IP

[asaraseka]

Terraform Version

Terraform v0.11.3
+ provider.google v1.7.0

Affected Resource(s)

Please list the resources as a list, for example:

• google_dataproc_cluster

Expected Behavior

Be able to access to IP addresses for created VMs (master and workers).
That would be useful if (like in our case) you want to create DNS records for created VMs

Actual Behavior

Now we can only get list of names for instances:
cluster_config.master_config.instance_names

Comments

I read that it will require some additional work, like trigger dataproc API to get VMs names, and then triggers Compute Engine API to get VMs info based on names.

[vijayboopathy-zz]

@danawillow @asaraseka I have a similar use case. Instance names wont help much if we do not know the connection strings for master.

[vgomenyuk]

Any updates here?

[BusiPlay]

Could you use a data resource here?

data "google_compute_instance" "dataproc_master" {
name = google_dataproc_cluser.dp_cluster.cluster_config.0.master_config.0.instance_names[0]
zone = "us-central1-a"
}

then reference data.google_compute_instance.dataproc_master.network_interface.network_ip in any DNS resources?

You could use a similar construct with a for_each for multiple instances.

[AlfatahB]

b/262220896

[AlfatahB]

I've looked into this issue and following are the details I have found out:

• google_dataproc_cluster is using the cluster_config.master_config.num_instances field to specify the number of master nodes to create. The names of these instances are fetched to the cluster_config.master_config.instance_names output-only field.
• It seems like this issue specifically wants to access the IP addresses of these created VMs so that these values can be used elsewhere in the terraform config file. e.g. to create DNS records for the VMs.

A probable solution can be to use the google_compute_instance datasource to get additional information about the instances of cluster_config.0.master_config.0.instance_names list. Such information can be used to create different resources. e.g. Here, we can use the IP address of VMs to create DNS records. An example of such a use case is given in this comment.

I've tried using the google_compute_instance datasource to get additional information about these VMs and it` s working fine. Just for the reference, the following is the terraform config file I've used:

resource "google_dataproc_cluster" "sample_cluster" {
  name     = "tf-test-dataproc"
  region   = "us-central1"

  cluster_config {
    master_config {
      num_instances = 3
      machine_type  = "n1-standard-2"  // can't be e2 because of min_cpu_platform
      disk_config {
        boot_disk_type    = "pd-ssd"
        boot_disk_size_gb = 35
      }
      min_cpu_platform = "Intel Skylake"
    }

    worker_config {
      num_instances = 3
      machine_type  = "n1-standard-2"  // can't be e2 because of min_cpu_platform
      disk_config {
        boot_disk_type    = "pd-standard"
        boot_disk_size_gb = 35
        num_local_ssds    = 1
      }

      min_cpu_platform = "Intel Broadwell"
    }

    preemptible_worker_config {
      num_instances = 1
      disk_config {
        boot_disk_type    = "pd-ssd"
        boot_disk_size_gb = 35
        num_local_ssds    = 1
      }
    }
  }
}

data "google_compute_instance" "dataproc_master" {
name = google_dataproc_cluster.sample_cluster.cluster_config.0.master_config.0.instance_names[0]
zone = "us-central1-b"
}