Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Add PullSecret controller to save pull secret data locally #425

Merged
merged 2 commits into from
Oct 8, 2024

Conversation

anik120
Copy link
Collaborator

@anik120 anik120 commented Oct 4, 2024

@anik120 anik120 requested a review from a team as a code owner October 4, 2024 13:44
@anik120
Copy link
Collaborator Author

anik120 commented Oct 4, 2024

Test performed:

Created:

  • test-namespace namespace
  • Additional required RBAC:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: manager-global-pull-secrets-role
  namespace: test-namespace
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: manager-global-pull-secrets-rolebinding
  namespace: test-namespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: manager-global-pull-secrets-role
subjects:
- kind: ServiceAccount
  name: catalogd-controller-manager
  namespace: olmv1-system
  • Secret
apiVersion: v1
data:
  .dockerconfigjson: ewogICJhdXRocy...
kind: Secret
metadata:
  name: pull-secret
  namespace: test-namespace
type: kubernetes.io/dockerconfigjson
  • ClusterCatalog with image hosted in a private registry (specifically the community-operators catalog)
apiVersion: olm.operatorframework.io/v1alpha1
kind: ClusterCatalog
metadata:
  name: redhat-community-operators
spec:
  priority: 0
  source:
    type: Image
    image:
      pollInterval: 24h
      ref: registry.redhat.io/redhat/community-operator-index:v4.18

Controller logs:

I1004 13:24:28.568725       1 secretsyncer_controller.go:93] "saved Secret data locally" controller="secret" controllerGroup="" controllerKind="Secret" Secret="test-namespace/pull-secret" namespace="test-namespace" name="pull-secret" reconcileID="b0ebfdba-c361-4a33-8f47-0bfaaa78d746" file="/tmp/catalogd/auth.json"
I1004 13:26:50.196943       1 cluster_catalog_webhook.go:24] "Invoking Default method for ClusterCatalog" logger="admission" webhookGroup="olm.operatorframework.io" webhookKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" resource={"group":"olm.operatorframework.io","version":"v1alpha1","resource":"clustercatalogs"} user="kubernetes-admin" requestID="94edaf65-10ce-4109-9e29-1cb095d63a9d" object={"kind":"ClusterCatalog","apiVersion":"olm.operatorframework.io/v1alpha1","metadata":{"name":"redhat-community-operators","creationTimestamp":null,"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"olm.operatorframework.io/v1alpha1\",\"kind\":\"ClusterCatalog\",\"metadata\":{\"annotations\":{},\"name\":\"redhat-community-operators\"},\"spec\":{\"priority\":0,\"source\":{\"image\":{\"pollInterval\":\"24h\",\"ref\":\"registry.redhat.io/redhat/community-operator-index:v4.18\"},\"type\":\"Image\"}}}\n"},"managedFields":[{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"olm.operatorframework.io/v1alpha1","time":"2024-10-04T13:26:50Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:priority":{},"f:source":{".":{},"f:image":{".":{},"f:pollInterval":{},"f:ref":{}},"f:type":{}}}}}]},"spec":{"source":{"type":"Image","image":{"ref":"registry.redhat.io/redhat/community-operator-index:v4.18","pollInterval":"24h0m0s"}},"priority":0},"status":{"lastUnpacked":null}}
I1004 13:26:50.197010       1 cluster_catalog_webhook.go:35] "default" logger="admission" webhookGroup="olm.operatorframework.io" webhookKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" resource={"group":"olm.operatorframework.io","version":"v1alpha1","resource":"clustercatalogs"} user="kubernetes-admin" requestID="94edaf65-10ce-4109-9e29-1cb095d63a9d" olm.operatorframework.io/metadata.name="redhat-community-operators" labels={"olm.operatorframework.io/metadata.name":"redhat-community-operators"}
I1004 13:26:50.202141       1 clustercatalog_controller.go:68] "reconcile starting" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" reconcileID="bf1dffa6-33e1-42c5-8350-8414a49b0a64"
I1004 13:26:50.206418       1 clustercatalog_controller.go:107] "reconcile ending" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" reconcileID="bf1dffa6-33e1-42c5-8350-8414a49b0a64"
I1004 13:26:50.206464       1 clustercatalog_controller.go:68] "reconcile starting" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" reconcileID="3414d5df-8740-4b1f-8f0c-79f26e26044e"
I1004 13:26:50.206503       1 main.go:231] "using available authentication information for pulling image" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" reconcileID="3414d5df-8740-4b1f-8f0c-79f26e26044e"
I1004 13:26:51.095799       1 containers_image.go:238] "no default policy found, using insecure policy" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" reconcileID="3414d5df-8740-4b1f-8f0c-79f26e26044e"
I1004 13:27:13.567042       1 containers_image.go:134] "pulled image" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" reconcileID="3414d5df-8740-4b1f-8f0c-79f26e26044e" ref="registry.redhat.io/redhat/community-operator-index:v4.18" digest="sha256:b62c5c97806fa27c87fdd7f184cd4b465ec19bdfc1e9be9434c1410eb2cced7f"
I1004 13:27:13.567766       1 containers_image.go:280] "unpacking image" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" reconcileID="3414d5df-8740-4b1f-8f0c-79f26e26044e" path="/var/cache/unpack/redhat-community-operators/sha256:b62c5c97806fa27c87fdd7f184cd4b465ec19bdfc1e9be9434c1410eb2cced7f"
I1004 13:27:14.453522       1 containers_image.go:292] "applied layer" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" reconcileID="3414d5df-8740-4b1f-8f0c-79f26e26044e" layer=0
I1004 13:27:14.939572       1 containers_image.go:292] "applied layer" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" reconcileID="3414d5df-8740-4b1f-8f0c-79f26e26044e" layer=1
I1004 13:27:15.046377       1 containers_image.go:292] "applied layer" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" reconcileID="3414d5df-8740-4b1f-8f0c-79f26e26044e" layer=2
I1004 13:27:15.821281       1 containers_image.go:292] "applied layer" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" reconcileID="3414d5df-8740-4b1f-8f0c-79f26e26044e" layer=3
I1004 13:27:16.332590       1 containers_image.go:292] "applied layer" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" reconcileID="3414d5df-8740-4b1f-8f0c-79f26e26044e" layer=4
I1004 13:27:16.600686       1 clustercatalog_controller.go:107] "reconcile ending" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" reconcileID="3414d5df-8740-4b1f-8f0c-79f26e26044e"
I1004 13:27:16.600734       1 clustercatalog_controller.go:68] "reconcile starting" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" reconcileID="9578c1c1-a12b-4c29-bb69-e713b33e428b"
I1004 13:27:16.600823       1 clustercatalog_controller.go:107] "reconcile ending" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="redhat-community-operators" namespace="" name="redhat-community-operators" reconcileID="9578c1c1-a12b-4c29-bb69-e713b33e428b"
I1004 13:45:54.221809       1 clustercatalog_controller.go:68] "reconcile starting" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="operatorhubio" namespace="" name="operatorhubio" reconcileID="f59b882b-8cf2-4368-9545-a7e09d3fd36d"
I1004 13:45:54.221928       1 main.go:231] "using available authentication information for pulling image" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="operatorhubio" namespace="" name="operatorhubio" reconcileID="f59b882b-8cf2-4368-9545-a7e09d3fd36d"
I1004 13:45:59.854326       1 containers_image.go:74] "image already unpacked" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="operatorhubio" namespace="" name="operatorhubio" reconcileID="f59b882b-8cf2-4368-9545-a7e09d3fd36d" ref="quay.io/operatorhubio/catalog:latest" digest="sha256:95338388fd5dcb766dd9aec30b05b4bb9b26f9072ae4cdceb2f6dc7ff3b8eaae"
I1004 13:46:00.403991       1 clustercatalog_controller.go:107] "reconcile ending" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="operatorhubio" namespace="" name="operatorhubio" reconcileID="f59b882b-8cf2-4368-9545-a7e09d3fd36d"
I1004 13:46:00.404046       1 clustercatalog_controller.go:68] "reconcile starting" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="operatorhubio" namespace="" name="operatorhubio" reconcileID="6995f438-43af-4bcc-b013-f4ee4c4a4110"
I1004 13:46:00.404131       1 clustercatalog_controller.go:107] "reconcile ending" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog" ClusterCatalog="operatorhubio" namespace="" name="operatorhubio" reconcileID="6995f438-43af-4bcc-b013-f4ee4c4a4110"

Log line to note from the Unpacker:

1004 13:26:50.206503       1 main.go:231] "using available authentication information for pulling image" logger="catalogd-controller" controller="clustercatalog" controllerGroup="olm.operatorframework.io" controllerKind="ClusterCatalog"

Final result 🎉 :

$ kubectl get clustercatalog                                                                
NAME                         LASTUNPACKED   SERVING   AGE
operatorhubio                3m22s          True      5m38s
redhat-community-operators   7s             True      33s

Copy link

codecov bot commented Oct 4, 2024

Codecov Report

Attention: Patch coverage is 24.24242% with 75 lines in your changes missing coverage. Please review.

Project coverage is 37.31%. Comparing base (8137da0) to head (2427c8a).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
cmd/manager/main.go 0.00% 46 Missing ⚠️
...nternal/controllers/core/pull_secret_controller.go 41.86% 21 Missing and 4 partials ⚠️
internal/source/containers_image.go 60.00% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #425      +/-   ##
==========================================
+ Coverage   35.28%   37.31%   +2.02%     
==========================================
  Files          14       15       +1     
  Lines         802      922     +120     
==========================================
+ Hits          283      344      +61     
- Misses        472      529      +57     
- Partials       47       49       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

err := r.Get(ctx, req.NamespacedName, secret)
if err != nil {
if apierrors.IsNotFound(err) {
logger.Info("secret not found")
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be a warn or error saying it will be unable to pull from pvt registries ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So there's no WARN for this particular logger we're using, only Info or Error. I don't think this needs to be an Error, especially since the function we're calling logs additional messages about the auth file being deleted.

@@ -105,6 +113,7 @@ func main() {
flag.StringVar(&keyFile, "tls-key", "", "The key file used for serving catalog contents over HTTPS. Requires tls-cert.")
flag.IntVar(&webhookPort, "webhook-server-port", 9443, "The port that the mutating webhook server serves at.")
flag.StringVar(&caCertDir, "ca-certs-dir", "", "The directory of CA certificate to use for verifying HTTPS connections to image registries.")
flag.StringVar(&globalPullSecret, "global-pull-secret", "", "The namespace/name of the global pull secret that is going to be used to pull bundle images.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this could be a bit confusing at a glance. One could read this as the namespace OR the name. Could format it like the error message below "/"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call, changed.

// https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md
err := os.WriteFile(r.AuthFilePath, dockerConfigJSON, 0600)
if err != nil {
return ctrl.Result{}, fmt.Errorf("failed to write secret data to file: %w", err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar comment here as Sid's below re: revealing file location

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the other ones, but I have the urge to keep this one. We'd want to see the specific error message for this when we get a must-gather (logs) for a bug report related to this.

@@ -49,12 +49,13 @@ func (i *ContainersImageRegistry) Unpack(ctx context.Context, catalog *catalogdv
return nil, reconcile.TerminalError(fmt.Errorf("error parsing catalog, catalog %s has a nil image source", catalog.Name))
}

srcCtx, err := i.SourceContextFunc(l)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

linter pointed this out, but no err check here

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops missed this, thanks for pointing it out. Added an error check below.

@anik120 anik120 force-pushed the secret-controller branch 5 times, most recently from 71bb7a0 to ad9c5b0 Compare October 7, 2024 13:36
@anik120 anik120 changed the title ✨ Add SecretSyncer controller to save pull secret data locally ✨ Add PullSecret controller to save pull secret data locally Oct 7, 2024
storageDir = "catalogs"
authFilePath = "/etc/catalogd/auth.json"
storageDir = "catalogs"
authFilePrefix = "catalogd-global-pull-secret.json"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this have the .json extension?

Copy link
Collaborator Author

@anik120 anik120 Oct 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see the confusion. It says "prefix" there but it's actually the suffix. Changing it. Also I just noticed the fmt.Sprintf already has the .json, thanks for pointing it out.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be fixed now. Here's what it looks like https://go.dev/play/p/FJlQgd_KS5R

@tmshort
Copy link
Contributor

tmshort commented Oct 7, 2024

Should this have changes that match operator-framework/operator-controller#1322 ?

@anik120
Copy link
Collaborator Author

anik120 commented Oct 8, 2024

Should this have changes that match operator-framework/operator-controller#1322 ?

@tmshort do you mean this? Am I missing any changes? I've been trying to keep both of them in sync and I don't see anything that I missed.

everettraven
everettraven previously approved these changes Oct 8, 2024
@everettraven everettraven added this pull request to the merge queue Oct 8, 2024
Merged via the queue into operator-framework:main with commit b1b145a Oct 8, 2024
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants