terraform-import failure: Cannot import non-existent remote object

[lens0021]

Terraform Version

Terraform v0.14.3
provider "registry.terraform.io/hashicorp/github" { version = "4.1.0" }

Affected Resource(s)

• github_repository

Terraform Configuration Files

terraform {
  required_version = "~> 0.14.0"

  backend "remote" {
    organization = "femiwiki"

    workspaces {
      name = "github"
    }
  }

  required_providers {
    github = {
      source  = "hashicorp/github"
      version = "~> 4.0"
    }
  }
}

provider "github" {
  organization = "femiwiki"
}

resource "github_repository" "legunto" {
  name = "legunto"
  ...
}

Full configuration: https://github.com/femiwiki/infra/blob/50eb0673787a8cab88041876632e45b6df6a2a57/github/repo.tf#L465-L472

Debug Output

https://gist.github.com/lens0021/35d5e8997a389bfcd91fa6cc976d2fbc

Expected Behavior

A state which is created based on https://github.com/femiwiki/legunto is Imported successfully.

Actual Behavior

"Error: Cannot import non-existent remote object" is printed. See the debug output for details.

Steps to Reproduce

1. Uncomment https://github.com/femiwiki/infra/blob/50eb0673787a8cab88041876632e45b6df6a2a57/github/repo.tf#L465-L472
2. terraform import github_repository.legunto legunto

Important Factoids

Our team is using Terraform Cloud (https://app.terraform.io/).

[olaven]

➕
Experiencing the same issue with version 4.4.0 and S3 backend.

[lens0021]

Same in Terraform v0.14.8 and terraform-provider-github v4.5.2.

[lens0021]

The name of the organization is empty in the log.

2021/04/09 08:35:52 [DEBUG] Reading repository: /legunto
2021/04/09 08:35:52 [DEBUG] Github API Request Details:
---[ REQUEST ]---------------------------------------
GET /repos//legunto HTTP/1.1

[lens0021]

TF_LOG=TRACE terraform import github_repository.legunto femiwiki/legunto also falied:

2021/04/09 08:44:00 [DEBUG] Reading repository: /femiwiki/legunto
2021/04/09 08:44:00 [DEBUG] Github API Request Details:
---[ REQUEST ]---------------------------------------
GET /repos//femiwiki/legunto HTTP/1.1

[kfcampbell]

Your debug output has the following snippet:

github_repository.legunto: Refreshing state... [id=legunto]
2021-01-02T03:48:07.191+0900 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: 2021/01/02 03:48:07 [DEBUG] Reading repository: /legunto
2021-01-02T03:48:07.191+0900 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: 2021/01/02 03:48:07 [DEBUG] Github API Request Details:
2021-01-02T03:48:07.191+0900 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: ---[ REQUEST ]---------------------------------------
2021-01-02T03:48:07.191+0900 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: GET /repos//legunto HTTP/1.1

in which we see id=legunto set correctly. However, the next line contains Reading repository: /legunto, with an appended slash, that we then see in the request string: GET /repos//legunto HTTP/1.1.

My guess is that somewhere we're accidentally appending an extra slash, and if we removed that, the GET request would succeed. I'm going to poke around the code a little bit to see if I can prove that.

[kfcampbell]

Nope, you're totally correct that it's indicative of a missing owner. I've been unable to reproduce this locally, which makes me think it's perhaps related to the remote backend organization configuration. I'm going to try to create a remote backend for this test and try to reproduce that way.

[kfcampbell]

I've been unable to repro this using Terraform Cloud, either...my full HCL is below:

resource "github_repository" "legunto" {
  name                 = "legunto"
  description          = "Fetch MediaWiki Scribunto modules from wikis"
  default_branch       = local.default_repo.default_branch
  has_issues           = local.default_repo.has_issues
  vulnerability_alerts = local.default_repo.vulnerability_alerts
  archive_on_destroy   = local.default_repo.archive_on_destroy
}

terraform {
  required_version = "~> 0.14.0"

  backend "remote" {
    organization = "example-org-b8a054"

    workspaces {
      name = "github"
    }
  }

  required_providers {
    github = {
      source  = "hashicorp/github"
      version = "~> 4.0"
    }
  }
}

provider "github" {
  organization = "kfcampbell-terraform-provider"
}

locals {
  default_repo = {
    # repository
    default_branch       = "main",
    has_issues           = true,
    vulnerability_alerts = true,
    archive_on_destroy   = true,

    # branch_protection
    pattern                         = "main"
    push_restrictions               = [],
    enforce_admins                  = false,
    dismiss_stale_reviews           = false,
    require_code_owner_reviews      = false,
    required_approving_review_count = 1,
  }
  with_cd = merge(local.default_repo, {
    # enforce_admins is temporarily disabled due to too few development members.
    enforce_admins = false,
  })
  docker = merge(local.default_repo, {
    # enforce_admins is temporarily disabled due to too few development members.
    enforce_admins = false,
    default_branch = "master",
    pattern        = "master"
  })
  bot = local.with_cd
}

The only things I've attempted to change from your setup is my GitHub organization name, and my Terraform Cloud organization name. I've also copied in your local configuration setup as well.

In order to run this test, I created a github workspace using my Terraform Cloud account, used terraform apply to create the resource, then destroyed the workspace to remove the terraform state, recreated the workspace, and then ran the below import command:

 sh$ terraform import github_repository.legunto legunto
github_repository.legunto: Importing from ID "legunto"...
github_repository.legunto: Import prepared!
  Prepared github_repository for import
github_repository.legunto: Refreshing state... [id=legunto]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

@lens0021 is there anything about this reproduction that appears incorrect or not in line with what you're doing?

[lens0021]

@kfcampbell Thank you for your investigation.

I retried the import in few other ways, I have noticed it was GITHUB_TOKEN that I was missing.

$ export GITHUB_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$ terraform import github_repository.legunto legunto
...
Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

It will be helpful if there could be a message for the missing token, Thanks!

[kfcampbell]

Oh I'm glad to hear you got it resolved! I think a 404 is somewhat appropriate if a token isn't present, since visiting a GitHub private repository from an unauthenticated or anonymous account will give a 404. I agree it's confusing to see the path missing the organization and perhaps the messaging could be improved.

[kungfuchicken]

still not resolved for me.
running TF 13.6 (because of reasons).
provider registry.terraform.io/hashicorp/github v4.10.1

[Artemu]

So for anyone having this issue.
You need to set the GITHUB_TOKEN in both TF Cloud and on your CLI. They also need to match - use the same token.

[call]

I've also been experiencing this issue, specifically when trying to import a github_repository resource into a child module.

If I run export GITHUB_OWNER=<org name>, the import works successfully.

[kohsuke]

I had the same symptom. Like @lens0021 , I saw in the debug output that the GET request to GitHub was missing the organization name part:

2021-07-21T16:37:32.222-0700 [DEBUG] provider.terraform-provider-github_v4.8.0: ---[ REQUEST ]---------------------------------------
2021-07-21T16:37:32.222-0700 [DEBUG] provider.terraform-provider-github_v4.8.0: GET /repos//demo HTTP/1.1
2021-07-21T16:37:32.222-0700 [DEBUG] provider.terraform-provider-github_v4.8.0: Host: api.github.com
2021-07-21T16:37:32.222-0700 [DEBUG] provider.terraform-provider-github_v4.8.0: User-Agent: go-github
2021-07-21T16:37:32.222-0700 [DEBUG] provider.terraform-provider-github_v4.8.0: Accept: application/vnd.github.scarlet-witch-preview+json, application/vnd.github.mercy-preview+json, application/vnd.github.baptiste-preview+json, application/vnd.github.nebula-preview+json
2021-07-21T16:37:32.222-0700 [DEBUG] provider.terraform-provider-github_v4.8.0: Accept-Encoding: gzip
2021-07-21T16:37:32.222-0700 [DEBUG] provider.terraform-provider-github_v4.8.0: 
2021-07-21T16:37:32.222-0700 [DEBUG] provider.terraform-provider-github_v4.8.0: 
2021-07-21T16:37:32.222-0700 [DEBUG] provider.terraform-provider-github_v4.8.0: -----------------------------------------------------

And just as he did, setting GITHUB_TOKEN environment variable resolved the issue.

What I'd love to see is the error message improvement.

[kfcampbell]

The original code and comment mentions that we shouldn't change the logic for how the configuration sets the owner without creating a new major release as it'd break backwards compatibility.

In the meantime, I've created this PR as a starting point to log which organization/owner is selected and therefore hopefully make it easier to see how the configuration is being interpreted. Do the folks on this thread see value/benefit in this? Feedback is appreciated.

[Olin-Vorechovsky]

IMHO the true cause of this is here:

terraform-provider-github/github/config.go

Lines 146 to 152 in 5a88727

	var owner Owner
	owner.v4client = v4client
	owner.v3client = v3client
	if c.Anonymous() {
	return &owner, nil
	} else {

When GITHUB_TOKEN env variable is not set (c.Anonymous() is true), the Owner.name property is not set.

Owner is meta interface{} passed into all CRUD methods and a lot of them depends on Owner.name as organization name.

[kfcampbell]

Oooh, good point. That is awfully confusing...in the absence of a total handling rewrite, I've added a few more logs in the PR that produce output such as:

(in an environment with both token and owner set)

2021/08/03 20:36:41 [DEBUG] Selecting owner kfcampbell-terraform-provider from GITHUB_OWNER environment variable
2021/08/03 20:36:41 [DEBUG] Token present; configuring authenticated owner: kfcampbell-terraform-provider

(in an environment in which the organization is set but the token is not)

2021/08/03 20:29:22 [DEBUG] Selecting owner kfcampbell-terraform-provider from GITHUB_ORGANIZATION environment variable
2021/08/03 20:29:22 [DEBUG] Selecting organization attribute as owner: kfcampbell-terraform-provider
2021/08/03 20:29:22 [DEBUG] No token present; configuring anonymous owner.

(for a user with neither owner nor token set)

2021/08/03 20:33:59 [DEBUG] Selecting owner  from GITHUB_OWNER environment variable
2021/08/03 20:33:59 [DEBUG] No token present; configuring anonymous owner.

It's still confusing since we're setting the owner variable in multiple different places that are used differently, but perhaps it's somewhat more clear than the current state. Thoughts?

[Olin-Vorechovsky]

To sum up:

• authentication is done if GITHUB_TOKEN is set
• organization name is setup by provider configuration property named owner
• there is at least one resource github_repository where there are two paths possible, one for organization, and one for authenticated user
• there is some try to handle wrong setup called checkOrganization() == !meta.(*Owner).IsOrganization == (owner.name == “”) but omitted for github_repository for apparent reason, and somehow missing in a lot of resources dependent on organization name (IMHO bug)
• I don't get up if there is some resources available for anonymous github client

Proposal:

• fmt.Errorf if GITHUB_TOKEN is empty to disable anonymous github client calls

This leads into unfortunate resource github_repository:

• missing provider.owner means act as authenticated user (by GITHUB_TOKEN) to create personal repository, but it gets in the way if we want to mix this with resources dependent on organization, they need provider.owner property filled!

[lubo]

Looks like at least owner and token arguments are completely ignored in the provider block. I have something like this:

Click to expand

terraform {
  required_providers {
    github = {
      source  = "integrations/github"
      version = "~> 4.0"
    }
  }
}

variable "github_token" {
  description = "A GitHub OAuth / Personal Access Token."
  sensitive   = true
  type        = string
}

provider "github" {
  owner = "ACME"
  token = var.github_token
}

Add this is logged:

2021-11-11T21:39:02.676+0100 [DEBUG] provider.terraform-provider-github_v4.18.0: 2021/11/11 21:39:02 [DEBUG] Selecting owner  from GITHUB_OWNER environment variable
2021-11-11T21:39:02.677+0100 [DEBUG] provider.terraform-provider-github_v4.18.0: 2021/11/11 21:39:02 [DEBUG] Setting write_delay_ms to 1000
2021-11-11T21:39:02.677+0100 [DEBUG] provider.terraform-provider-github_v4.18.0: 2021/11/11 21:39:02 [DEBUG] No token present; configuring anonymous owner.

However, all works as expected when I use GITHUB_OWNER and GITHUB_TOKEN environmental variables. I'm using Terraform v1.0.10.

[gnarea]

Setting GITHUB_TOKEN and GITHUB_OWNER in the CLI worked for me too. However, GITHUB_TOKEN doesn't have to be the one set in Terraform Cloud -- in fact, you really shouldn't do it for security reasons. Just create a GitHub PAT under your own account with the read-only public_repo scope if it's public (otherwise you're going to have to give it the repo scope).

[github-actions]

👋 Hey Friends, this issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Please add the Status: Pinned label if you feel that this issue needs to remain open/active. Thank you for your contributions and help in keeping things tidy!

[crmne]

I'm having the same issue with the latest version of this provider (5.18.3) and a repo internal to my org (myorg/mobile). I set TF_LOG=INFO and this is the relevant part of the log:

$ terraform import module.github.github_repository.repos[\"mobile\"] mobile
...
2023-03-25T17:32:17.042+0100 [WARN]  Provider "registry.terraform.io/integrations/github" produced an unexpected new value for module.github.github_repository.repos["mobile"] during refresh.
      - Root resource was present, but now absent
2023-03-25T17:32:17.042+0100 [ERROR] vertex "import module.github.github_repository.repos[\"mobile\"] result" error: Cannot import non-existent remote object
2023-03-25T17:32:17.042+0100 [ERROR] vertex "module.github.github_repository.repos[\"mobile\"] (import id \"mobile\")" error: Cannot import non-existent remote object
2023-03-25T17:32:17.042+0100 [ERROR] vertex "module.github.github_repository.repos (expand)" error: Cannot import non-existent remote object
╷
│ Error: Cannot import non-existent remote object
│ 
│ While attempting to import an existing object to "module.github.github_repository.repos[\"mobile\"]", the provider detected that no object exists with the given
│ id. Only pre-existing objects can be imported; check that the id is correct and that it is associated with the provider's configured region or endpoint, or use
│ "terraform apply" to create a new remote object for this resource.

[coyotespike]

I am having the same issue now as well, on the same version as @crmne

It seem to succeed (Import prepared!) and then utterly fails.

It also refuses to work with a nested module, although the nested module is able to create repos from scratch.

github_repository.eaas: Importing from ID "eaas"...
github_repository.eaas: Import prepared!
  Prepared github_repository for import
github_repository.eaas: Refreshing state... [id=eaas]
╷
│ Error: Cannot import non-existent remote object
│ 
│ While attempting to import an existing object to "github_repository.eaas",
│ the provider detected that no object exists with the given id. Only
│ pre-existing objects can be imported; check that the id is correct and that
│ it is associated with the provider's configured region or endpoint, or use
│ "terraform apply" to create a new remote object for this resource.
╵

[coyotespike]

I have now solved this problem.

Several issues may have contributed.

• I may or may not have started with the now-deprecated hashicorp/github. You must ensure with terraform -version or terraform providers that only one provider is present. If more than is present, remove it with terraform state replace-provider registry.terraform.io/hashicorp/github registry.terraform.io/integrations/github.

• The docs for this provider note that a nested module must have a providers import there as well, not just at the root. Otherwise everything breaks horribly. Go into your subdirectories and put this in providers.tf or something:

# this must be included in each submodule or you will get horrible and strange errors
terraform {
  required_providers {
    github = {
      source  = "integrations/github"
      version = "~> 5.0"
    }
  }
}

• My module is nested. You must provide a path to it. terraform import -config=github/repos github_repository.embeddings_api embeddings_api, where github and repos are both directories.

• Finally, the resource was still not properly created. After import, Terraform wanted to destroy and to create the resource. Of course Github will not allow it to do so. Looking at terraform.tfstate, the new resource (my imported repo) was missing this at the top: "module": "module.repos",. All the other blocks had it. I manually added it and things worked. I had to do this for every imported resource.

god what a journey. Hope this helps and good luck.

[crmne]

Thanks for your comments @coyotespike, glad that it worked for you but unfortunately they didn't help in my case.

However, I managed to find the issue, by setting TF_LOG=DEBUG:

2023-03-27T12:57:34.686+0200 [DEBUG] provider.terraform-provider-github_v5.18.3: 2023/03/27 12:57:34 [DEBUG] GitHub API Request Details:
2023-03-27T12:57:34.686+0200 [DEBUG] provider.terraform-provider-github_v5.18.3: ---[ REQUEST ]---------------------------------------
2023-03-27T12:57:34.686+0200 [DEBUG] provider.terraform-provider-github_v5.18.3: GET /repos//freshflow.ai HTTP/1.1
2023-03-27T12:57:34.686+0200 [DEBUG] provider.terraform-provider-github_v5.18.3: Host: api.github.com

the GET should be done on /repos/freshflowai/freshflow.ai but I can't find a way in the resource to specify the owner of the repository.

[crmne]

I managed to solve the issue for me. It looks like I didn't set GITHUB_TOKEN locally, and was assuming it would be using Terraform Cloud's GITHUB_TOKEN. My bad, but perhaps it would be helpful to have a message, at least at the TF_LOG=INFO that clearly tells the use they haven't set GITHUB_TOKEN. Thanks!

[tekumara]

I'm getting this error when using

provider "github" {
  owner = "tekumara"
  app_auth {
  }
}

And setting the env vars:

GITHUB_APP_ID
GITHUB_APP_INSTALLATION_ID
GITHUB_APP_PEM_FILE

With this authentication I can plan and apply, but not import.

If I switch to GITHUB_TOKEN (ie: not app auth) it works.