Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkg/asset/ignition/bootstrap: Add /root/.docker/config.json #775

Merged

Conversation

wking
Copy link
Member

@wking wking commented Dec 3, 2018

And export REGISTRY_AUTH_FILE pointing at that file. We already provide /opt/tectonic/manifests/pull.json with a Kubernetes secret that contains the auth JSON, but the new file has the auth JSON without any wrapping. This resolves issues with auth-protected update payloads:

[core@ip-10-0-10-189 ~]$ journalctl -n5 -u bootkube.service
-- Logs begin at Mon 2018-12-03 06:54:37 UTC, end at Mon 2018-12-03 07:24:52 UTC. --
Dec 03 07:24:48 ip-10-0-10-189 bootkube.sh[696]: Trying to pull quay.io/openshift-release-dev/ocp-v4.0@sha256:69bae91f6a933045175170cebd1caf00b8216a096fd4d402c41957a99aa3435b...Failed
Dec 03 07:24:48 ip-10-0-10-189 bootkube.sh[696]: unable to pull quay.io/openshift-release-dev/ocp-v4.0@sha256:69bae91f6a933045175170cebd1caf00b8216a096fd4d402c41957a99aa3435b: unable to pull image: Error determining manifest MIME type for docker://quay.io/openshift-release-dev/ocp-v4.0@sha256:69bae91f6a933045175170cebd1caf00b8216a096fd4d402c41957a99aa3435b: Error reading manifest sha256:69bae91f6a933045175170cebd1caf00b8216a096fd4d402c41957a99aa3435b in quay.io/openshift-release-dev/ocp-v4.0: unauthorized: access to the requested resource is not authorized
Dec 03 07:24:48 ip-10-0-10-189 systemd[1]: bootkube.service: main process exited, code=exited, status=125/n/a
Dec 03 07:24:48 ip-10-0-10-189 systemd[1]: Unit bootkube.service entered failed state.
Dec 03 07:24:48 ip-10-0-10-189 systemd[1]: bootkube.service failed.

The environment variable is documented here.

I've also renamed the manifest template variable to PullSecretBase64 to distinguish it from the unencoded PullSecret I'm adding to the bootstrap template parameters. These are different template parameter sets, but I think recycling the same name would be unnecessarily confusing.

@openshift-ci-robot openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Dec 3, 2018
@wking wking force-pushed the pull-secret-for-bootkube.sh branch from 0099514 to dfbf1ef Compare December 3, 2018 08:18
@wking
Copy link
Member Author

wking commented Dec 3, 2018

Before this lands, we'll want to make sure it fixes CI for #773.

@openshift-ci-robot openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 3, 2018
@wking wking changed the title pkg/asset/ignition/bootstrap: Add /opt/tectonic/pull-auth.json pkg/asset/ignition/bootstrap: Add /root/.docker/config.json Dec 3, 2018
@wking wking force-pushed the pull-secret-for-bootkube.sh branch from dfbf1ef to d273367 Compare December 3, 2018 18:11
@openshift-ci-robot openshift-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Dec 3, 2018
@wking
Copy link
Member Author

wking commented Dec 3, 2018

@smarterclayton, we can run CI on this and secret images by cherry-picking it onto #773 (which is what I'm doing now), but it would be good to be running CI for "can we launch from private images?" in master without hacks like that. Can we get pull-secret protections for at least one constituent image on the CI registry? Maybe the release payload? Or is that not easy with installer CI using RELEASE_IMAGE_LATEST since openshift/release#1793? Maybe the tooling that pushes RELEASE_IMAGE_LATEST can push it up as a private image?

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Dec 3, 2018

@wking: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
ci/prow/e2e-libvirt dfbf1efac26254c8b78ab426f8149106ae690b3b link /test e2e-libvirt

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@wking
Copy link
Member Author

wking commented Dec 3, 2018

e2e-aws failed to generate the bootstrap-complete event:

level=warning msg="Failed to connect events watcher: Get https://ci-op-8z2f4xly-1d3f3-api.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/kube-system/events?watch=true: dial tcp 100.24.138.105:6443: connect: connection refused"
level=fatal msg="Error executing openshift-install: waiting for bootstrap-complete: timed out waiting for the condition"

I'll investigate in a fresh CI cluster:

/retest

We already provide /opt/tectonic/manifests/pull.json with a Kubernetes
secret that contains the auth JSON, but the new file has the auth JSON
without any wrapping.  This resolves issues with auth-protected update
payloads [1]:

  [core@ip-10-0-10-189 ~]$ journalctl -n5 -u bootkube.service
  -- Logs begin at Mon 2018-12-03 06:54:37 UTC, end at Mon 2018-12-03 07:24:52 UTC. --
  Dec 03 07:24:48 ip-10-0-10-189 bootkube.sh[696]: Trying to pull quay.io/openshift-release-dev/ocp-v4.0@sha256:69bae91f6a933045175170cebd1caf00b8216a096fd4d402c41957a99aa3435b...Failed
  Dec 03 07:24:48 ip-10-0-10-189 bootkube.sh[696]: unable to pull quay.io/openshift-release-dev/ocp-v4.0@sha256:69bae91f6a933045175170cebd1caf00b8216a096fd4d402c41957a99aa3435b: unable to pull image: Error determining manifest MIME type for docker://quay.io/openshift-release-dev/ocp-v4.0@sha256:69bae91f6a933045175170cebd1caf00b8216a096fd4d402c41957a99aa3435b: Error reading manifest sha256:69bae91f6a933045175170cebd1caf00b8216a096fd4d402c41957a99aa3435b in quay.io/openshift-release-dev/ocp-v4.0: unauthorized: access to the requested resource is not authorized
  Dec 03 07:24:48 ip-10-0-10-189 systemd[1]: bootkube.service: main process exited, code=exited, status=125/n/a
  Dec 03 07:24:48 ip-10-0-10-189 systemd[1]: Unit bootkube.service entered failed state.
  Dec 03 07:24:48 ip-10-0-10-189 systemd[1]: bootkube.service failed.

I think the ~/.docker directory name is unfortunate, since we're not
actually using Docker at all.  But ~/.docker/config.json is an auth
location shared by Podman [2], CRI-O [3], and the kubelet [4].  And
it's the most generic on-disk location that will survive reboots (/run
will not [5]).

I've also renamed the manifest template variable to PullSecretBase64
to distinguish it from the unencoded PullSecret I'm adding to the
bootstrap template parameters.  These are different template parameter
sets, but I think recycling the same name would be unnecessarily
confusing.

[1]: openshift#773 (comment)
[2]: https://github.com/containers/libpod/blob/v0.11.1.1/docs/podman-pull.1.md#options
[3]: https://github.com/kubernetes-sigs/cri-o/blob/master/vendor/github.com/containers/image/pkg/docker/config/config.go#L32
[4]: https://kubernetes.io/docs/concepts/containers/images/#configuring-nodes-to-authenticate-to-a-private-registry
[5]: http://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s15.html#runPurpose
@wking wking force-pushed the pull-secret-for-bootkube.sh branch from d273367 to 791a41c Compare December 3, 2018 19:41
@wking
Copy link
Member Author

wking commented Dec 3, 2018

Issue was a missing .template suffix (I keep forgetting about that :p). Fixed with d273367 -> 791a41c.

@smarterclayton
Copy link
Contributor

smarterclayton commented Dec 3, 2018 via email

@crawford
Copy link
Contributor

crawford commented Dec 3, 2018

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 3, 2018
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: crawford, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot merged commit 6585f5d into openshift:master Dec 3, 2018
wking added a commit to wking/openshift-installer that referenced this pull request Dec 3, 2018
Through 6585f5d (Merge pull request openshift#775 from
wking/pull-secret-for-bootkube.sh, 2018-12-03).
@wking wking deleted the pull-secret-for-bootkube.sh branch December 3, 2018 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants