Bug: ArtifactRegistryRepository errors with NotReady even after the registry has already been created. #348

karlkfi · 2020-12-15T19:08:22Z

Describe the bug
ArtifactRegistryRepository errors with NotReady even after the registry has already been created (as seen in UI or CLI).

status:
  conditions:
  - lastTransitionTime: "2020-12-09T23:33:01Z"
    message: "Update call failed: error applying desired state: summary: Error updating
      Repository \"projects/platform-project/locations/us-central1/repositories/platform\":
      googleapi: got HTTP response code 404 with body: <!DOCTYPE html>\n<html lang=en>\n
      \ <meta charset=utf-8>\n  <meta name=viewport content=\"initial-scale=1, minimum-scale=1,
      width=device-width\">\n  <title>Error 404 (Not Found)!!1</title>\n  <style>\n
      \   *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7%!a(MISSING)uto
      0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png)
      100%!p(MISSING)x no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a
      img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png)
      no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
      no-repeat 0%/100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
      0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
      no-repeat;-webkit-background-size:100%}}#logo{display:inline-block;height:54px;width:150px}\n
      \ </style>\n  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\n
      \ <p><b>404.</b> <ins>That’s an error.</ins>\n  <p>The requested URL <code>/v1beta1/projects/platform-dev-div1-envb/locations/us-central1/repositories/platform?alt=json</code>
      was not found on this server.  <ins>That’s all we know.</ins>\n, detail: "
    reason: UpdateFailed
    status: "False"
    type: Ready

ConfigConnector Version
1.27.1

To Reproduce
Steps to reproduce the behavior:

apiVersion: artifactregistry.cnrm.cloud.google.com/v1beta1
kind: ArtifactRegistryRepository
metadata:
  name: platform
  namespace: platform-namespace
spec:
  format: DOCKER
  location: us-central1
  description: Docker registry

caieo · 2020-12-16T17:54:59Z

Hi @karlkfi, thank you for raising this issue! I was able to reproduce your 404 error yesterday and was able to work around it by deleting the k8s object for ARRepository and then reapplying it. (By reapplying it, you're basically acquiring the existing ARRepository). We will investigate why the 404 error appears when trying to create an ARRepository for the first time and will post updates to this thread.

jcanseco · 2021-02-09T22:09:55Z

Hi @karlkfi, are you still encountering this issue? I am currently unable to reproduce this issue on either KCC 1.27.1 or the latest version of KCC.

jcanseco · 2021-02-10T20:55:39Z

Discussed internally and this doesn't seem to be a problem anymore. Closing for now, but will reopen if the issue re-surfaces.

karlkfi · 2021-02-10T21:04:34Z

deleting and re-creating the resources seemed to make it work. not quite sure why that was required after KCC upgrade tho.

karlkfi · 2021-02-11T01:28:05Z

The bug came back...

KCC: 1.37.0

status:
  conditions:
  - lastTransitionTime: "2021-02-10T21:16:48Z"
    message: "Update call failed: error applying desired state: summary: Error updating
      Repository \"projects/platform-dev-exp-envb-9392/locations/us-central1/repositories/boutique\":
      googleapi: got HTTP response code 404 with body: <!DOCTYPE html>\n<html lang=en>\n
      \ <meta charset=utf-8>\n  <meta name=viewport content=\"initial-scale=1, minimum-scale=1,
      width=device-width\">\n  <title>Error 404 (Not Found)!!1</title>\n  <style>\n
      \   *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7%!a(MISSING)uto
      0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png)
      100%!p(MISSING)x no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a
      img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png)
      no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
      no-repeat 0%/100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
      0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
      no-repeat;-webkit-background-size:100%}}#logo{display:inline-block;height:54px;width:150px}\n
      \ </style>\n  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\n
      \ <p><b>404.</b> <ins>That’s an error.</ins>\n  <p>The requested URL <code>/v1beta1/projects/platform-dev-exp-envb-9392/locations/us-central1/repositories/boutique?alt=json</code>
      was not found on this server.  <ins>That’s all we know.</ins>\n, detail: "
    reason: UpdateFailed
    status: "False"
    type: Ready
  createTime: "2021-02-10T20:56:49.390081Z"
  name: boutique
  updateTime: "2021-02-10T20:56:49.390081Z"

jcanseco · 2021-02-11T01:35:35Z

Ok I see -- I'll try to reproduce again (this time letting the repo live for a few hours and see if I see the same issue).

jcanseco · 2021-02-11T02:31:59Z

I'm finally able to reproduce the issue, and what's more, it seems like just updating the resource at all triggers the error. Will continue to investigate.

jcanseco · 2021-02-11T02:43:32Z

Ok I think there is just something wrong with the update logic, and the resource triggers the issue eventually after resource creation due to KCC updating the labels periodically as part of the leasing mechanism.

So for now, while we investigate the root cause, our recommendation is to disable the leasing mechanism for your ArtifactRegistryRepository resources. That should ensure the resource stays Ready as long as the spec isn't updated and there is no drift.

The leasing mechanism can be disabled by adding the annotation:

annotations:
  cnrm.cloud.google.com/management-conflict-prevention-policy: none

Note that you'll need to abandon and re-acquire any existing ArtifactRegistryRepository resources for this workaround to work.

jcanseco · 2021-03-12T00:18:51Z

This issue has now been fixed in v1.39.0.

karlkfi added the bug Something isn't working label Dec 15, 2020

caieo mentioned this issue Dec 16, 2020

Error 404 updating ArtifactRegistryRepository #339

Closed

jcanseco closed this as completed Feb 10, 2021

jcanseco reopened this Feb 11, 2021

jcanseco mentioned this issue Feb 11, 2021

Fix bug: artifact_registry_repository update always failing GoogleCloudPlatform/magic-modules#4482

Merged

5 tasks

This was referenced Feb 17, 2021

Fix bug: artifact_registry_repository update always failing hashicorp/terraform-provider-google-beta#2968

Merged

Fix bug: artifact_registry_repository update always failing hashicorp/terraform-provider-google#8491

Merged

jcanseco closed this as completed Mar 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug: ArtifactRegistryRepository errors with NotReady even after the registry has already been created. #348

Bug: ArtifactRegistryRepository errors with NotReady even after the registry has already been created. #348

karlkfi commented Dec 15, 2020 •

edited

Loading

caieo commented Dec 16, 2020

jcanseco commented Feb 9, 2021

jcanseco commented Feb 10, 2021

karlkfi commented Feb 10, 2021

karlkfi commented Feb 11, 2021 •

edited

Loading

jcanseco commented Feb 11, 2021

jcanseco commented Feb 11, 2021

jcanseco commented Feb 11, 2021 •

edited

Loading

jcanseco commented Mar 12, 2021

Bug: ArtifactRegistryRepository errors with NotReady even after the registry has already been created. #348

Bug: ArtifactRegistryRepository errors with NotReady even after the registry has already been created. #348

Comments

karlkfi commented Dec 15, 2020 • edited Loading

caieo commented Dec 16, 2020

jcanseco commented Feb 9, 2021

jcanseco commented Feb 10, 2021

karlkfi commented Feb 10, 2021

karlkfi commented Feb 11, 2021 • edited Loading

jcanseco commented Feb 11, 2021

jcanseco commented Feb 11, 2021

jcanseco commented Feb 11, 2021 • edited Loading

jcanseco commented Mar 12, 2021

karlkfi commented Dec 15, 2020 •

edited

Loading

karlkfi commented Feb 11, 2021 •

edited

Loading

jcanseco commented Feb 11, 2021 •

edited

Loading