google_dataflow_job doesn't pick up "project" variable

[MaxBinnewies]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

Terraform v0.11.8

• provider.google v1.18.0
• provider.kubernetes v1.2.0
• provider.null v1.0.0

Affected Resource(s)

• google_dataflow_job

Terraform Configuration Files

resource "google_dataflow_job" "account-backup" {
  project           = "751024901768"
  depends_on        = ["google_project_service.dataflow-service", "google_storage_bucket.backup-bucket"]
  name              = "pubsub-account-backup"
  template_gcs_path = "gs://dataflow-templates/latest/Cloud_PubSub_to_GCS_Text"
  temp_gcs_location = "gs://accounts-backup-bucket2/tmp/"
  zone              = "asia-east1"

  parameters {
    inputTopic           = "projects/${var.project}/topics/${var.inputTopic}"
    outputFilenamePrefix = "accountData-"
    outputDirectory      = "gs://accounts-backup-bucket2/logs/"
  }
}

Debug Output

https://gist.github.com/MaxBinnewies/b72d781d860dda5ebd0c721c71bd048f

Panic Output

Expected Behavior

Dataflow Job to be created in project 751024901768

Actual Behavior

Terraform tries to create a Dataflow job in a different project of mine 1063229073055. This is the project that the Service account I am using belongs to. I used this Service account to create the correct project 751024901768 also through Terraform.
The error message is correct in the sense that the other project does not have the Dataflow API enabled. However, the project where it is actually supposed to go does, I can create the job manually without issue.
However, Terraform does not seem to be picking up the "project" variable and tries to create the Dataflow job in the wrong project. The same applies if I set "project" to

• "751024901768"
• "${var.project}" -> which resolves to "jih200f0ih-dds"
• "jih200f0ih-dds"
• "projects/jih200f0ih-dds"

As you can see Terraform outputs the projectId correctly, but then without any other errors in between, throws an error for a different project, leading me to believe that this might be a bug.

Steps to Reproduce

1. terraform apply

Important Factoids

The used service account belongs to project 1063229073055 and has been used to create project 751024901768.

References

• #0000

[paddycarver]

The project is getting picked up there, I believe the way services work on the API is you must have enabled the API on the project that the service account making the request belongs to. If you can share debug logs, we can probably prove that.

[breznik]

I'm experiencing the exact same issue - and in my case, the API is not enabled in the service account making the request (e.g. dedicated project/service account for terraform), but is enabled in the target project. Similarly, the project is resolving to the correct project, but the error is for the project id belonging to the project I have terraform admin running in.

[paddycarver]

That's just how services work in GCP, I believe. I'm going to close this out, because I don't know that there's anything for us to do, and I think upstream believes this is working as intended. The fix is to enable the service on the project the service account making the request belongs to.

[breznik]

I'm quite confused by that response. So, let's say I have two projects: terraform-admin & awesome-project. The service account in the terraform-admin project has access to enable/disable apis and manage infrastructure in awesome-project. When I try to create a dataflow job for awesome-project, it fails because it's trying to create the dataflow job in terraform-admin project, even though the plan resolved to use awesome-project. That is the bug I believe the OP was referring to, and the same issue I ran into. Effectively, dataflow job support in terraform doesn't work the same as all the other gcp resources do.

[morgante]

When I try to create a dataflow job for awesome-project, it fails because it's trying to create the dataflow job in terraform-admin project, even though the plan resolved to use awesome-project.

Do you have logs of this happening? That would indeed be a serious bug.

In reality what I think you're seeing is:

1. You have Service Account credentials from the terraform-admin project
2. You create a Terraform resource meant for awesome-project
3. When Terraform goes to create the resource, it is indeed targeting awesome-project but because your credentials come from terraform-admin GCP checks that the Dataflow API is enabled on terraform-admin. GCP often requires that a specific API is enabled on the project your credentials are coming from and the project you want to create the resource in.
4. If you enabled the Dataflow API on both projects, it should successfully create the job in the awesome-project.

[breznik]

Ah ok - it was point #3 that you listed which I ran into. Wasn't aware it needed the API enabled in both projects. Thanks for clarifying!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!