Add the "google_project" datasource

[eilgin]

Allow the possibility for one user to fetch the project details (especially the project_id) based on a given name.
To retrieve the wished values, we'll only return the latest active project that matched the name.

The use case for this datasource to exist is when one resource is referencing a pre-existing one. For example, a user is creating a project and want to create a peering connection with an existing network. At the moment it would have to hardcode the project id. I propose a more elegant possibility:

data "google_project" "existing_project" {
  name = "my-project"
}

data "google_compute_network" "existing_network" {
  name = "default-us-east1"
  project = "${data.google_project.existing_project.project_id}"
}

resource "google_compute_network" "default" {
  name                                  = "foobar"
  auto_create_subnetworks = "false"
}

resource "google_compute_network_peering" "peering1" {
  name               = "peering1"
  network          = "${google_compute_network.default.self_link}"
  peer_network = "${data.google_compute_network.existing_network.self_link}"
}

resource "google_compute_network_peering" "peering2" {
  name               = "peering2"
  network          = "${data.google_compute_network.existing_network.self_link}"
  peer_network = "${google_compute_network.default.self_link}"
}

For my use case, i've created a bastion machine in a dedicated project which must be created before other machines and therefore this PR would greatly alleviate the problem of referencing this existing project.

[rosbo]

Hi Thomas,

Since multiple project can have the same name, I am hesitant about adding this as it would lead to unexpected behavior if two projects share the same display name.

I can see some value in having a data source to retrieve the name and the project number from the project id. However, I don't think the display name or the project number is ever needed for setting up other terraform resource. It could be used if a user want to include the project display name in the description of another resource or something like that.

You can create a variable in your schema for the project id and pass that one around instead of using this data source.

Thoughts?

[eilgin]

Well this datasource is clearly stating that it will fetched the latest active project if 2 or more projects hold the same name so the result will be consistent. Besides, holding the project_id of the existing project (in my case i've called it "bastion") in a variable is what i'm already doing and it's very cumbersome since i'm using a module to spawn new projects (which creates the peering connection with the "bastion" project) so everytime there's a change, i'll have to manually update this variable which defeat the purpose of automating the infra.

I'm confident that this datasource would be useful for other cases but it's up to you to accept it or not 😉

[rosbo]

My concern is that if anyone in your org create a new project named "bastion" (without knowing this is effectively a "protected" project name), then it will silently take this new project in all your configs since this newly created project will be the most recent. When you have a large org with lots of developers, it is easy for someone to create a project called "bastion" and silently break things. This is why I am hesitant in encouraging this pattern. I am curious to see what @danawillow or @paddycarver think. Maybe I am being paranoid :)

You said:

everytime there's a change, i'll have to manually update this variable which defeat the purpose of automating the infra.

What "change" are you referring too?
And why would the project id of the bastion project change every time there is a change?

[danawillow]

I agree with @rosbo- I'm not totally opposed, but I think we'd need giant warnings on our docs about this. I'm also definitely curious for more details about the workflow that makes this data source necessary.

[paddycarver]

I also am hesitant about this, and would be really interested in hearing more about the problem you're trying to solve, so we could see if there's another solution. You mentioned that you're creating a project in a module, and you want to set up a peering connection to the bastion project. I'd love to see which part of the config you need to manually update, and why, and see if we can't help interpolate that.

[eilgin]

@rosbo my bad. i didn't think about using this datasource in the context of a big corp infra. Even tough it would be rare that someone would create the exact same project name within the same org. (or the same folder), it could happen and this would definitively break one's plan.

In my personal use case, i'm using a script to "terraform" my infra. it will:

1. (only once) apply plan for the "bootstrap" project (the one that was given in the "provider") to set things like the project services
2. (only once) apply plan for the "bastion" project which will create some resources, fetch its "project_id" (via terraform output ...) and write it to a (versioned) file inside the "standard_project" module.
3. apply plan for the other projects that will use this module.

Inside this module, i declare a "template_file":

// retrieve the bastion project id to link project network to it
data "template_file" "bastion_project_id" {
  template = "$${chomp(file("${path.module}/assets/$${prefix}bastion_project_id.txt"))}"

  vars {
    prefix = "${var.env == "test" ? "test_" : ""}"
  }
}

This template will be used to fetch the bastion network self_link:

data "google_compute_network" "bastion" {
  name    = "bastion-network"
  project = "${data.template_file.bastion_project_id.rendered}"
}

Finally, i use the previous data to create the peering connection:

resource "google_compute_network_peering" "peering_to_bastion" {
  name         = "peering-${local.full_name}-${random_id.suffix.dec}-to-bastion"
  network      = "${google_compute_network.default.self_link}"
  peer_network = "${data.google_compute_network.bastion.self_link}"

  lifecycle {
    create_before_destroy = true
  }
}

resource "google_compute_network_peering" "peering_from_bastion" {
  name         = "peering-${local.full_name}-${random_id.suffix.dec}-from-bastion"
  network      = "${data.google_compute_network.bastion.self_link}"
  peer_network = "${google_compute_network.default.self_link}"

  lifecycle {
    create_before_destroy = true
  }
}

Using this new datasource would allow me to directly reference the latest active "bastion" project more easily (using an "hardcoded" project_id feels like a kludge).

One solution could be to rename this datasource google_last_active_project removing any ambiguity about its purpose. What do you think ? But if you think that this datasource doesn't belong to this provider as it doesn't solve enough use cases and decides not to include it, it'd be fine by me (i'll just stick to my current solution and be a little disappointed 😭).

[rosbo]

Assuming you are storing your state file remotely, have you considered using terraform_remote_state?

1. Your bastion project saves his state either any backend supported by Terraform in a bucket called "bastion". (Like TF enterprise or GCS).
2. Your bastion project outputs the its project id.
3. Each of your project could get the project id from the remote state ${data.terraform_remote_state.bastion.project_id}

[eilgin]

Thank you so much @rosbo !
I didn't know about this provider and since i'm using the "gcs" backend state, i'm able to query the "bastion" root module and fetch its project id.

Well, in this case this PR isn't relevant anymore.

[rosbo]

I'm glad that this solution works for you!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!