feature request: Add support for Cloud Dataproc Workflow Templates

[luckeyca]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

Please add new resource or module to the new dataproc workflow templates api. This will help build an end-to-end fully automated dataproc pipeline from within terraform itself which, I believe, will be greatly appreciated by the users on big data platform.

feature description from google: https://cloud.google.com/dataproc/docs/concepts/workflows/overview

New or Affected Resource(s)

• google_dataproc_cluster
• google_dataproc_job
• google_dataproc_cluster_iam_policy
• google_dataproc_cluster_iam_binding
• google_dataproc_job_iam_member

Potential Terraform Configuration

# Propose what you think the configuration to take advantage of this feature should look like.
# We may not use it verbatim, but it's helpful in understanding your intent.

should be able to do all the functions with the api as listed below:

https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.workflowTemplates
https://cloud.google.com/sdk/gcloud/reference/beta/dataproc/workflow-templates/

sample workflow using command line and api

https://cloud.google.com/dataproc/docs/concepts/workflows/using-workflows

this workflow can use existing dataproc or create/destroy on-demand.

References

• b/179479331

[emilymye]

should be able to do all the functions with the api as listed below:

Do you expect to be able to instantiate the workflow template from Terraform? I can easily see how to CRUD a workflow template as a resource, but starting a workflow from a template seems like more the work of a provisioner/CI. (I guess I'm saying if we do this it's likely going to follow the normal Terraform resource functionality for apply/refresh/destroy)

[jaketf]

I think it would be very useful to define workflow templates in terraform.
Agreed, instantiating can be done by provisioner / CI or by an application.
However, being able to build out a suite of workflow templates that are linked w/ other terraform resources ie. network, service accounts etc. would be helpful as workflow templates include cluster definition.

[preston-hf]

Triggering the templates should be left to a provisioner, but it would be quite nice to be able to define them with terraform

[xiaobaitusi]

This resource is also requested by one of ConfigConnector customers.

[ocervell]

Hello, any updates here ?

[rileykarson]

We'll need to cut a new DCL release, but looks like we can attempt using tpgtools here.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!