Support custom service account in Dataflow job resource

[migibert]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

It would be great if we could specify a custom service account during Dataflow job creation, in order to define fine grained permissions on buckets, bigquery datasets, etc...

The API already supports this: https://cloud.google.com/dataflow/security-and-permissions#controller-service-account

And good news, Dataflow Go lib also supports it: https://godoc.org/google.golang.org/api/dataflow/v1b3#Environment

New or Affected Resource(s)

• google_dataflow_job

Potential Terraform Configuration

resource "google_storage_bucket" "mybucket" {
    name = "mybucket"
    force_destroy = true
}

resource "google_service_account" "dataflow-sa" {
  account_id   = "dataflow-sa"
  display_name = "DataFlow Service Account"
}

resource "google_storage_bucket_iam_member" "dataflow-gcs" {
  bucket = "${google_storage_bucket.mybucket.name}"
  role   = "roles/storage.objectAdmin"
  member = "serviceAccount:${google_service_account.dataflow-sa.email}"
}

resource "google_project_iam_member" "dataflow-worker" {
  role   = "roles/dataflow.worker"
  member = "serviceAccount:${google_service_account.dataflow-sa.email}"
}

resource "google_dataflow_job" "big_data" {
    name = "dfjob-test-%s"
    template_gcs_path = "gs://dataflow-templates/wordcount/template_file"
    temp_gcs_location = "${google_storage_bucket.mybucket.url}"

    parameters {
        inputFile = "gs://dataflow-samples/shakespeare/kinglear.txt"
        output    = "${google_storage_bucket.mybucket.url}/output"
    }
    zone = "us-central1-f"
    project = "%s"
    service_account_email = "${google_service_account.dataflow-sa.email}"
    on_delete = "cancel"
}

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!