github_branch_default *always* returns 422 Visibility is already private. [] on deletion

[kiddom-kq]

I've found an issue that seems to crop up on repo deletion.
The issue appears to be with private repos that have a `` resource set.

Terraform Version

This issue is reproducible on the latest versions. (as of today; 2020-12-04)

❯ terraform version --json
{
  "terraform_version": "0.14.0",
  "terraform_revision": "",
  "provider_selections": {
    "registry.terraform.io/hashicorp/aws": "3.19.0",
    "registry.terraform.io/hashicorp/github": "4.1.0"
  },
  "terraform_outdated": false
}

Affected Resource(s)

• github_branch_default

Terraform Configuration Files

Here is a simple test-case that is identical to the test case I used to find/document this bug. I have stripped out org/personal information and substituted generic tokens in place:

terraform {

  backend "s3" {
   # omitted...
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3"
    }
    github = {
      source = "hashicorp/github"
      version = "~> 4"
    }
  }
}

provider "github" {
  token = var.GITHUB_TOKEN
  organization = var.GITHUB_ORGANIZATION
}

resource "github_repository" "myOrg_testing" {
  name = "gh-issue-test"
  visibility = "private"

  has_issues = true
  has_projects = false
  has_wiki = false
  is_template = false
  allow_merge_commit = true
  allow_squash_merge = true
  allow_rebase_merge = true
  delete_branch_on_merge = false
  has_downloads = false
  auto_init = false
  archived = false
  vulnerability_alerts = true
  gitignore_template = ""
  license_template = ""
}

# See: https://github.com/terraform-providers/terraform-provider-github/blob/master/website/docs/r/branch_default.html.markdown
resource "github_branch_default" "myOrg_testing" {
  repository = github_repository.myOrg_testing.name
  branch     = "MyNonMainDefaultBranch"
}

resource "github_branch_protection" "myOrg_testing" {
  repository_id = github_repository.myOrg_testing.node_id
  pattern       = "MyNonMainDefaultBranch"
  enforce_admins = false
}

#####
# Access Control
#####
resource "github_team_repository" "testing-ops-team-access" {
  team_id    = github_team.ops.id
  repository = github_repository.myOrg_testing.name
  permission = "admin"
}

# Rest of org
resource "github_team_repository" "testing-developers-team-access" {
  team_id    = github_team.developers.id
  repository = github_repository.myOrg_testing.name
  permission = "push"
}

Debug Output

Please provide a link to a GitHub Gist containing the complete debug output: https://www.terraform.io/docs/internals/debugging.html. Please do NOT paste the debug output in the issue; just paste a link to the Gist.

Here is a short version:

❯ terraform destroy --target=github_branch_default.myOrg_testing
github_branch_default.myOrg_testing: Refreshing state... [id=gh-issue-test]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # github_branch_default.myOrg_testing will be destroyed
  - resource "github_branch_default" "myOrg_testing" {
      - branch     = "MyNonMainDefaultBranch" -> null
      - id         = "gh-issue-test" -> null
      - repository = "gh-issue-test" -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.


Warning: Resource targeting is in effect

You are creating a plan with the -target option...<snip>

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

github_branch_default.myOrg_testing: Destroying... [id=gh-issue-test]

Warning: Applied changes may be incomplete

The plan was created with the -target option in effect...<snip>

Note that the -target option is not suitable for routine use...<snip>

Error: PATCH https://api.github.com/repos/myOrg/gh-issue-test: 422 Visibility is already private. []

Note: I get the same result if I comment out only the github_branch_default resource and then run a normal terraform apply. I just used --target to hone which resource caused the error.

I am more than happy to provide the full TF_LOG=TRACE output, but it's 3K lines long with the --target flag. I've included the relevant bits below:

2020-12-04T09:34:24.880-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: -----------------------------------------------------
2020-12-04T09:34:24.880-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: 2020/12/04 09:34:24 [TRACE] Acquiring lock for GitHub API request (%!q(<nil>))
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: 2020/12/04 09:34:25 [TRACE] Releasing lock for GitHub API request (%!q(<nil>))
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: 2020/12/04 09:34:25 [DEBUG] Github API Response Details:
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: ---[ RESPONSE ]--------------------------------------
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: HTTP/1.1 422 Unprocessable Entity
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: Content-Length: 93
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: Access-Control-Allow-Origin: *
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: Content-Security-Policy: default-src 'none'
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: Content-Type: application/json; charset=utf-8
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: Date: Fri, 04 Dec 2020 17:34:25 GMT
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: Server: GitHub.com
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: Status: 422 Unprocessable Entity
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: Vary: Accept-Encoding, Accept, X-Requested-With
2020-12-04T09:34:25.306-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: X-Accepted-Oauth-Scopes:
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: X-Content-Type-Options: nosniff
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: X-Frame-Options: deny
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: X-Github-Media-Type: github.baptiste-preview; format=json, github.nebula-preview; format=json
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: X-Github-Request-Id: E775:1715:749DA3:8FD7A3:5FCA7320
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: X-Oauth-Scopes: admin:org, delete_repo, repo
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: X-Ratelimit-Limit: 5000
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: X-Ratelimit-Remaining: 4997
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: X-Ratelimit-Reset: 1607106864
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: X-Ratelimit-Used: 3
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: X-Xss-Protection: 1; mode=block
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4:
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: {
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4:  "message": "Visibility is already private.",
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4:  "documentation_url": "https://github.com/pricing"
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: }
2020-12-04T09:34:25.307-0800 [DEBUG] plugin.terraform-provider-github_v4.1.0_x4: -----------------------------------------------------
2020/12/04 09:34:25 [DEBUG] github_branch_default.myOrg_testing: apply errored, but we're indicating that via the Error pointer rather than returning it: PATCH https://api.github.com/repos/myOrg/gh-issue-test: 422 Visibility is already private. []
2020/12/04 09:34:25 [TRACE] EvalWriteState: writing current state object for github_branch_default.myOrg_testing
2020/12/04 09:34:25 [TRACE] vertex "github_branch_default.myOrg_testing (destroy)": visit complete
2020/12/04 09:34:25 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/github\"] (close)" errored, so skipping
2020/12/04 09:34:25 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2020/12/04 09:34:25 [TRACE] dag/walk: upstream of "root" errored, so skipping

Panic Output

No panic produced.

Expected Behavior

I would expect the default branch configuration to be deleted w/o the 422 API error.

Actual Behavior

I get this error returned from GH API:

Error: PATCH https://api.github.com/repos/myOrg/gh-issue-test: 422 Visibility is already private. []

Steps to Reproduce

1. (see note2 below) comment out the github_branch_default resource in the above 'test case' configuration.
2. run tf apply against the config and observe that 4 resources are created, one of which is the repo.
3. un-comment out the github_branch_default resource in the above 'test-case' config.
4. use either the web UI or git checkout -b && git push to make the branch called MyNonMainDefaultBranch in the new repo
5. run tf apply against the config and observe that one new resource will be created
6. determine that the repo and associated resources created in the test config must now be deleted
7. comment out all the resources in the above test case
8. run tf apply against the config and observe that 5 resources will be destroyed
9. observe that some resources have been deleted:

• github_team_repository.testing-ops-team-access: Destruction complete after 0s
• github_team_repository.testing-developers-team-access: Destruction complete after 2s
• github_branch_protection.myOrg_testing: Destruction complete after 3s

10. (see note1 below). Manually remove the github_branch_default resource from the state file
11. run tf apply against the config and observe that the remaining resource (the github_repository.myOrg_testing ) will be destroyed

Note1:: If, instead of manually removing the github_branch_default resource from the state file as in step 10, you leave ONLY the github_repository resource un-commented, terraform will still only destroy the github_branch_default.myOrg_testing resource. This tells me that the dependency resolution is logically correct but inefficient/un-optimized; the deletion of the github_branch_default does not matter if the github_repository is to be deleted. Deleting the repo necessarily deletes the default branch config.

Note2: Failure to comment out the github_branch_default resource in step 1 will result in an error:

Error: PATCH https://api.github.com/repos/myOrg/gh-issue-test: 422 Validation Failed [{Resource:Repository Field:default_branch Code:invalid Message:Cannot update default branch for an empty repository. Please init the repository and push first.}]

Important Factoids

I can reproduce this behavior with the token that was allocated specifically for our automated Terraform ci/cd pipeline and with my personal token which has 'Owner' level permissions attached to it.

The workaround is to:

1. comment out / delete the github_branch_default resource from the *.tf file(s).

2. manually remove the github_branch_default resource from the state file: terraform state rm github_branch_default.myOrg_testing:

Removed github_branch_default.myOrg_testing
Successfully removed 1 resource instance(s).

3. re-run terraform apply and let it observe that the github_repository is missing and thus should be deleted. Deleting the repo necessarily deletes the repos' default branch config.

References

These issues seem related:

https://github.com/terraform-providers/terraform-provider-github/issues/580

[onurg]

just hit the same issue on a terratest run:

• we're creating a dummy repo with visibility set to private
• we apply, test and destroy
• on destroy it fails

TestRepositoryValidity 2021-01-25T19:58:34Z logger.go:66: module.repositories["repositories/test-dummy-repo-1"].github_branch_default.default_branch: Destroying... [id=test-dummy-repo-1]
TestRepositoryValidity 2021-01-25T19:58:35Z logger.go:66: 
TestRepositoryValidity 2021-01-25T19:58:35Z logger.go:66: Error: PATCH https://api.github.com/repos/xxxx/test-dummy-repo-1: 422 Visibility is already private. []
TestRepositoryValidity 2021-01-25T19:58:35Z logger.go:66: 
TestRepositoryValidity 2021-01-25T19:58:35Z logger.go:66: 
TestRepositoryValidity 2021-01-25T19:58:35Z retry.go:80: Returning due to fatal error: FatalError{Underlying: error while running command: exit status 1; 
Error: PATCH https://api.github.com/repos/xxxx/test-dummy-repo-1: 422 Visibility is already private. []

}
    destroy.go:11: 
        	Error Trace:	destroy.go:11
        	            				terratest.go:56
        	            				tests_test.go:50
        	Error:      	Received unexpected error:
        	            	FatalError{Underlying: error while running command: exit status 1; 
        	            	Error: PATCH https://api.github.com/repos/xxxx/test-dummy-repo-1: 422 Visibility is already private. []
        	            	
        	            	}

[jcudit]

Seems like the fix released in #666 only corrected the create / update operations. We'll need to follow up with the delete operation to finalize the fix. Thanks for reporting @onurg.

Can anyone else confirm the v4.3.1 release is not sufficient here?

[onurg]

hi @jcudit @kiddom-kq, just verified v4.3.1 fixes it. if somebody else also wants to confirm that the fix was sufficient, feel free to do so.

here's the destroy output from the same run I posted prior:

TestRepositoryValidity 2021-01-26T13:24:28-05:00 retry.go:72: terraform [destroy -auto-approve -input=false -lock=false]
TestRepositoryValidity 2021-01-26T13:24:28-05:00 logger.go:66: Running command terraform with args [destroy -auto-approve -input=false -lock=false]
TestRepositoryValidity 2021-01-26T13:24:29-05:00 logger.go:66: module.teams["teams/dummy-team"].github_team.create_team: Refreshing state... [id=4465638]
TestRepositoryValidity 2021-01-26T13:24:29-05:00 logger.go:66: module.teams["teams/dummy-team"].github_team_membership.add_member["Whitening-McClean"]: Refreshing state... [id=4465638:Whitening-McClean]
TestRepositoryValidity 2021-01-26T13:24:29-05:00 logger.go:66: module.repositories["repositories/test-dummy-repo-1"].github_repository.create_repo: Refreshing state... [id=test-dummy-repo-1]
TestRepositoryValidity 2021-01-26T13:24:29-05:00 logger.go:66: module.repositories["repositories/test-dummy-repo-1"].github_team_repository.team_membership["dummy-team"]: Refreshing state... [id=4465638:test-dummy-repo-1]
TestRepositoryValidity 2021-01-26T13:24:29-05:00 logger.go:66: module.repositories["repositories/test-dummy-repo-1"].github_branch_default.default_branch: Refreshing state... [id=test-dummy-repo-1]
TestRepositoryValidity 2021-01-26T13:24:30-05:00 logger.go:66: module.repositories["repositories/test-dummy-repo-1"].github_repository_file.codeowners: Refreshing state... [id=test-dummy-repo-1/.github/CODEOWNERS]
TestRepositoryValidity 2021-01-26T13:24:31-05:00 logger.go:66: module.repositories["repositories/test-dummy-repo-1"].github_team_repository.team_membership["dummy-team"]: Destroying... [id=4465638:test-dummy-repo-1]
TestRepositoryValidity 2021-01-26T13:24:31-05:00 logger.go:66: module.repositories["repositories/test-dummy-repo-1"].github_repository_file.codeowners: Destroying... [id=test-dummy-repo-1/.github/CODEOWNERS]
TestRepositoryValidity 2021-01-26T13:24:31-05:00 logger.go:66: module.repositories["repositories/test-dummy-repo-1"].github_team_repository.team_membership["dummy-team"]: Destruction complete after 1s
TestRepositoryValidity 2021-01-26T13:24:33-05:00 logger.go:66: module.repositories["repositories/test-dummy-repo-1"].github_repository_file.codeowners: Destruction complete after 3s
TestRepositoryValidity 2021-01-26T13:24:33-05:00 logger.go:66: module.repositories["repositories/test-dummy-repo-1"].github_branch_default.default_branch: Destroying... [id=test-dummy-repo-1]
TestRepositoryValidity 2021-01-26T13:24:34-05:00 logger.go:66: module.repositories["repositories/test-dummy-repo-1"].github_branch_default.default_branch: Destruction complete after 1s
TestRepositoryValidity 2021-01-26T13:24:34-05:00 logger.go:66: module.repositories["repositories/test-dummy-repo-1"].github_repository.create_repo: Destroying... [id=test-dummy-repo-1]
TestRepositoryValidity 2021-01-26T13:24:36-05:00 logger.go:66: module.repositories["repositories/test-dummy-repo-1"].github_repository.create_repo: Destruction complete after 1s
TestRepositoryValidity 2021-01-26T13:24:36-05:00 logger.go:66: module.teams["teams/dummy-team"].github_team_membership.add_member["Whitening-McClean"]: Destroying... [id=4465638:Whitening-McClean]
TestRepositoryValidity 2021-01-26T13:24:37-05:00 logger.go:66: module.teams["teams/dummy-team"].github_team_membership.add_member["Whitening-McClean"]: Destruction complete after 2s
TestRepositoryValidity 2021-01-26T13:24:37-05:00 logger.go:66: module.teams["teams/dummy-team"].github_team.create_team: Destroying... [id=4465638]
TestRepositoryValidity 2021-01-26T13:24:39-05:00 logger.go:66: module.teams["teams/dummy-team"].github_team.create_team: Destruction complete after 1s
TestRepositoryValidity 2021-01-26T13:24:39-05:00 logger.go:66:
TestRepositoryValidity 2021-01-26T13:24:39-05:00 logger.go:66: Destroy complete! Resources: 6 destroyed.