Using gce_cluster_config -> service_account_scopes in google_dataproc_cluster always recreate the cluster

[kdzhao]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

Terraform v0.11.14

• provider.google v2.12.0
• provider.google-beta v2.12.0
• provider.vault v2.1.0

Affected Resource(s)

• google_dataproc_cluster

Expected Behavior

If there is no change to cluster_config -> gce_cluster_config -> service_account_scopes, the cluster should not be recreated.

Actual Behavior

The cluster is recreated even if there is no change for the service_account_scopes.

Steps to Reproduce

Just set the value, for example:
service_account_scopes = ["sql-admin"]

• #0000

[rileykarson]

Hey @kdzhao! Can you share the terraform plan result you've received? This should be canonicalised correctly.

[kdzhao]

It looks like service_account_scopes indeed is the issue:
when I specify it, the number of it reduces from 4 to 1.

The terraform apply output (related to service account scope) looks like:

  cluster_config.0.gce_cluster_config.0.service_account_scopes.#:          "4" => "1" (forces new resource)
  cluster_config.0.gce_cluster_config.0.service_account_scopes.1328717722: "https://www.googleapis.com/auth/devstorage.read_write" => "" (forces new resource)
  cluster_config.0.gce_cluster_config.0.service_account_scopes.172152165:  "https://www.googleapis.com/auth/logging.write" => "" (forces new resource)
  cluster_config.0.gce_cluster_config.0.service_account_scopes.3804780973: "https://www.googleapis.com/auth/cloud.useraccounts.readonly" => "" (forces new resource)
  cluster_config.0.gce_cluster_config.0.service_account_scopes.4205865871: "https://www.googleapis.com/auth/sqlservice.admin" => "https://www.googleapis.com/auth/sqlservice.admin"

And the related Terraform code is like:

gce_cluster_config {
  zone                   = "us-central1-a"
  service_account_scopes = ["sql-admin"]
  ....

[kdzhao]

And a bit search, I wonder if Terraform/Google provider counts the default scopes as mentioned in this doc:
https://cloud.google.com/compute/docs/access/service-accounts
So when I specify one, it gets confused with the default ones and believes there is a change of them, when I run the same code again.

[rileykarson]

Ah- per https://www.terraform.io/docs/providers/google/r/dataproc_cluster.html#service_account_scopes, those scopes are required to be specified on the resource. I'll make that message more explicit. We don't have much ability to allow a user to only specify a subset of the scopes, so unfortunately if any extra scopes are defined those need to be added too.

[kdzhao]

Oh, I see. After adding all the scopes as below, I see now it works as expected.

  service_account_scopes = ["https://www.googleapis.com/auth/cloud.useraccounts.readonly", "https://www.googleapis.com/auth/devstorage.read_write", "https://www.googleapis.com/auth/logging.write", "https://www.googleapis.com/auth/sqlservice.admin"]

Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!