[feature request] Allow secondary-workers to be non_preemptible in google_dataproc_cluster

[carlosmarin]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

Based on resource_dataproc_cluster.go#L817-L823, it looks like the Dataproc secondary workers always have to be preemptible and there is no way to specify them as non-preemptible, since machine_type cannot be set for the preemptible_worker_config. It would be nice to allow a non_preemptible_worker_config

New or Affected Resource(s)

Terraform v0.13.5
provider.google v3.48.0

• google_dataproc_cluster

Potential Terraform Configuration

Secondary workers seem to automatically be: preemptible_worker_config, with that convention, maybe there could be an optional, mutually-exclusive, non_preemptible_worker_config that uses the same or a similar schema as worker_config.

Otherwise, a property like:

secondary_worker_config.preemptible = false

and another construct for secondary-workers that isn't preemptible_worker_config.

References

• https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dataproc_cluster#example-usage---advanced

[rileykarson]

Note that there may be assumptions about the schema of preemptible_worker_config from when it was always preemptible.

We also likely don't wan't to rename preemptible_worker_config since renames of objects are pretty complex- we can add a note to the docs that it's an outdated name.

[aharelick]

There is an undocumented workaround by creating a cluster with "dataproc:secondary-workers.preemptible" = "false" as a property in the override_properties section of the software_config.

[andrew-murphy-inmar]

This might not have to be (as big of) a breaking change, if the "is_preemptible" attribute in preemptible_worker_config were optional and defaulted to "true". All existing code would continue to work - the only conflict would occur if the undocumented software config property mentioned above were also set to a different value.

[tsamaras]

I went ahead and tried to implement this with the existing naming scheme. It works fine as long as you aren't too offended by the idea of non-preemptible preemptible workers. I'll post the PR to magic-modules shortly

[c2thorn]

Was this closed in GoogleCloudPlatform/magic-modules#5686 ?

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.