replace kube-core rendering with openshift operators render #420

mfojtik · 2018-10-05T14:03:32Z

This change moves us away from kube-core renderer and use the kube-apiserver, kube-controller-manager and kube-scheduler operator renderers to gather all necessary bootstrap manifests.

@abhinavdahiya @smarterclayton this is what we want and we should make working.

mfojtik · 2018-10-05T14:06:57Z

FYI: Depends on openshift/cluster-kube-scheduler-operator#16 openshift/cluster-kube-controller-manager-operator#33 openshift/cluster-kube-controller-manager-operator#32 and openshift/cluster-kube-apiserver-operator#52

mfojtik · 2018-10-05T14:16:04Z

/hold

mfojtik · 2018-10-08T12:21:21Z

pkg/asset/ignition/bootstrap/bootstrap.go

-	defaultReleaseImage = "registry.svc.ci.openshift.org/openshift/origin-release:v4.0"
+	rootDir = "/opt/tectonic"
+
+	// TODO: This should be decided by installer, not hard-coded


@smarterclayton how are we supposed to handle this? :-) Also what is the preferred registry for our images? The RHCOS seems to have RH registry hardcoded as default, so using the :latest tag does not work and we really don't want to pin this into stable tag, otherwise we won't be able to test the changes in our operator images....

I guess we want all images in quay.io at some point but we want a knob to switch that to registry.svc.ci.openshift.org for the CI?

abhinavdahiya · 2018-10-08T16:45:50Z

so the kube-core-operator installs these things in addition to api

1. kube-proxy
2. kube-dns
3. pod-checkpointer

What is the plan for these?

derekwaynecarr · 2018-10-10T03:30:01Z

/cc @sjenning we need to discuss who shall install pod-checkpointer

openshift-ci-robot · 2018-10-10T03:30:04Z

@derekwaynecarr: GitHub didn't allow me to request PR reviews from the following users: need, install, who, shall, we, to, discuss.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @sjenning we need to discuss who shall install pod-checkpointer

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mfojtik · 2018-10-10T08:13:11Z

kube-proxy

kube-dns

/cc @ironcladlou

mfojtik · 2018-10-10T08:21:57Z

pkg/asset/tls/kubeletcertkey.go

@@ -29,7 +29,9 @@ func (a *KubeletCertKey) Generate(dependencies asset.Parents) error {
 	dependencies.Get(kubeCA)

 	cfg := &CertCfg{
-		Subject:      pkix.Name{CommonName: "system:serviceaccount:kube-system:default", Organization: []string{"system:serviceaccounts:kube-system"}},
+		// system:masters is a hack to get the kubelet up without kube-core
+		// TODO(node): make kubelet bootstrapping secure with minimal permissions eventually switching to system:node:* CommonName


/cc @sjenning

deads2k · 2018-10-17T20:27:57Z

/test all

mfojtik · 2018-10-17T20:42:12Z

/retest

mfojtik · 2018-10-17T20:43:46Z

@abhinavdahiya reported that issue here: https://github.com/openshift/release/issues/1955

abhinavdahiya · 2018-10-17T21:41:38Z

the e2e is failing here

Error: Error applying plan:

6 error(s) occurred:

* module.vpc.aws_eip.nat_eip[5]: 1 error(s) occurred:

* aws_eip.nat_eip.5: Error creating EIP: AddressLimitExceeded: The maximum number of addresses has been reached.
	status code: 400, request id: c512d767-2493-4812-9561-5d1197723cf7
* module.vpc.aws_eip.nat_eip[0]: 1 error(s) occurred:

* aws_eip.nat_eip.0: Error creating EIP: AddressLimitExceeded: The maximum number of addresses has been reached.
	status code: 400, request id: cac50c99-211c-49fc-9204-1c9e99f03ae7
* module.vpc.aws_eip.nat_eip[3]: 1 error(s) occurred:

* aws_eip.nat_eip.3: Error creating EIP: AddressLimitExceeded: The maximum number of addresses has been reached.
	status code: 400, request id: 77877ad8-d126-4850-894c-01e31861aa08
* module.vpc.aws_eip.nat_eip[4]: 1 error(s) occurred:

* aws_eip.nat_eip.4: Error creating EIP: AddressLimitExceeded: The maximum number of addresses has been reached.
	status code: 400, request id: 4ab4d76f-ae87-41d5-b51e-4905882fe710
* module.vpc.aws_eip.nat_eip[1]: 1 error(s) occurred:

* aws_eip.nat_eip.1: Error creating EIP: AddressLimitExceeded: The maximum number of addresses has been reached.
	status code: 400, request id: 952b9a6f-5742-4d1d-87ea-44f5b80d796e
* module.vpc.aws_eip.nat_eip[2]: 1 error(s) occurred:

* aws_eip.nat_eip.2: Error creating EIP: AddressLimitExceeded: The maximum number of addresses has been reached.
	status code: 400, request id: aaaed3f9-64d7-4d82-975c-81eeda85da7e

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

I am going to ask @wking to cleanup the CI account. :(

abhinavdahiya · 2018-10-17T22:39:25Z

/test e2e-aws

abhinavdahiya · 2018-10-17T22:53:53Z

/retest

wking · 2018-10-17T23:13:40Z

The previous e2e-aws error was:

2018/10/17 22:42:09 Executing template e2e-aws
2018/10/17 22:42:09 Creating or restarting template instance
2018/10/17 22:42:09 Waiting for template instance to be ready
2018/10/17 22:42:12 Ran for 2m5s
error: could not run steps: could not wait for template instance to be ready: could not determine if template instance was ready: failed to create objects: object is being deleted: pods "e2e-aws" already exists

That's openshift/release#1955.

deads2k · 2018-10-17T23:38:05Z

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_installer/420/pull-ci-openshift-installer-master-e2e-aws/781

/tmp/openshift/build-rpms/rpm/BUILD/origin-4.0.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:684
Expected error:
    <*errors.errorString | 0xc420a0f160>: {
        s: "failed to get logs from pod-secrets-69066412-d263-11e8-bde6-0a58ac1018e2 for secret-volume-test: an error on the server (\"unknown\") has prevented the request from succeeding (get pods pod-secrets-69066412-d263-11e8-bde6-0a58ac1018e2)",
    }
    failed to get logs from pod-secrets-69066412-d263-11e8-bde6-0a58ac1018e2 for secret-volume-test: an error on the server ("unknown") has prevented the request from succeeding (get pods pod-secrets-69066412-d263-11e8-bde6-0a58ac1018e2)
not to have occurred
/tmp/openshift/build-rpms/rpm/BUILD/origin-4.0.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:2351

@sttts @mfojtik maybe we hadn't gotten this far before?

deads2k · 2018-10-17T23:59:52Z

https://github.com/openshift/origin/pull/21286/files merged. Missing the operator half maybe?

openshift-bot · 2018-10-18T09:01:04Z

/retest

Please review the full test history for this PR and help us cut down flakes.

bootkube: override control-plane configs – fix kubectl logs

sttts · 2018-10-18T09:29:22Z

/lgtm

openshift-ci-robot · 2018-10-18T09:29:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, mfojtik, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [abhinavdahiya]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sttts · 2018-10-18T09:31:19Z

pkg/asset/ignition/bootstrap/content/bootkube.go

+apiVersion: kubecontrolplane.config.openshift.io/v1
+kind: KubeAPIServerConfig
+kubeletClientInfo:
+  ca: "" # kubelet uses self-signed serving certs. TODO: fix kubelet pki


@sjenning this is for you. We have self-signed kubelet serving certs right now. I.e. the kube-apiserver->kubelet communication is insecure because the kubelet identity is not checked.

This is the old kube-core behaviour btw. So we are just copying that here, but should do better. I read @LiGgit commenting somewhere that up to kube 1.10 we don't support CSRs for properly signed kubelet serving certs. Has this changed upstream in 1.11 or 1.12?

sttts · 2018-10-18T09:38:54Z

pkg/asset/ignition/bootstrap/content/bootkube.go

+	KubeControllerManagerConfigOverridesTemplate = template.Must(template.New("kube-controller-manager-config-overrides.yaml").Parse(`
+apiVersion: kubecontrolplane.config.openshift.io/v1
+kind: KubeControllerManagerConfig
+`))


@squeed this file is also for you. You can override any setting related to networking here, like cluster CIDR for example.

Catching up with c9b0e2f (manifests: stop using kube core operator, 2018-10-08, openshift#420). Generated with: $ dep ensure using: $ dep version dep: version : v0.5.0 build date : git hash : 22125cf go version : go1.10.3 go compiler : gc platform : linux/amd64 features : ImportDuringSolve=false

Catching up with c9b0e2f (manifests: stop using kube core operator, 2018-10-08, openshift#420).

Adds ClusterK8sIO from e2dc955 (pkg/asset: add ClusterK8sIO, machines.Worker assets, 2018-10-15, openshift#468) and Master from 586ad45 (pkg/asset: Add asset for Master machines, 2018-10-18, openshift#491). Removes KubeCoreOperator from c9b0e2f (manifests: stop using kube core operator, 2018-10-08, openshift#420). Generated with: $ openshift-install graph | dot -Tsvg >docs/design/resource_dep.svg using: $ dot -V dot - graphviz version 2.30.1 (20170916.1124)

openshift-ci-robot requested review from rajatchopra and steveej October 5, 2018 14:04

openshift-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 5, 2018

mfojtik force-pushed the remove-kco-render branch from d9904e6 to a2884e2 Compare October 5, 2018 14:04

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 5, 2018

mfojtik force-pushed the remove-kco-render branch from a2884e2 to 3209d52 Compare October 8, 2018 10:28

openshift-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 8, 2018

mfojtik commented Oct 8, 2018

View reviewed changes

mfojtik force-pushed the remove-kco-render branch 6 times, most recently from 04a64c9 to f2e6925 Compare October 8, 2018 16:03

openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 9, 2018

openshift-ci-robot requested a review from sjenning October 10, 2018 03:30

mfojtik force-pushed the remove-kco-render branch from 17f6a43 to 0d1dede Compare October 10, 2018 08:07

openshift-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 10, 2018

openshift-ci-robot requested a review from ironcladlou October 10, 2018 08:13

mfojtik commented Oct 10, 2018

View reviewed changes

mfojtik force-pushed the remove-kco-render branch from 21eed3e to c43391c Compare October 10, 2018 08:35

openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 10, 2018

mfojtik force-pushed the remove-kco-render branch from 7dbd330 to 7687262 Compare October 10, 2018 12:06

bootkube: add installer config overrides for the control plane

6a1b896

Merge pull request #9 from sttts/sttts-installer-overrides

54e2880

bootkube: override control-plane configs – fix kubectl logs

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Oct 18, 2018

openshift-ci-robot assigned sttts Oct 18, 2018

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 18, 2018

sttts reviewed Oct 18, 2018

View reviewed changes

openshift-merge-robot merged commit ace571b into openshift:master Oct 18, 2018

This was referenced Oct 18, 2018

kubelet uses self-signed serving certs #486

Closed

remove temporary bootkube kube-proxy and kube-dns manifests #487

Closed

remove temporary bootkube pod checkpointer manifests #488

Closed

wking mentioned this pull request Oct 20, 2018

vendor: Drop kube-core #506

Merged

wking added a commit to wking/openshift-installer that referenced this pull request Oct 20, 2018

pkg/asset/manifests/tectonic: Drop KubeCoreOperatorImage

1b0d0f6

Catching up with c9b0e2f (manifests: stop using kube core operator, 2018-10-08, openshift#420).

wking mentioned this pull request Oct 20, 2018

docs/design/resource_dep: Rebuild after kube-core removal #508

Merged

wking mentioned this pull request Dec 5, 2018

libvirt: Add Terraform variables for memory/CPU, bump master to 4GiB #785

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

replace kube-core rendering with openshift operators render #420

replace kube-core rendering with openshift operators render #420

mfojtik commented Oct 5, 2018

mfojtik commented Oct 5, 2018

mfojtik commented Oct 5, 2018

mfojtik Oct 8, 2018

abhinavdahiya commented Oct 8, 2018

derekwaynecarr commented Oct 10, 2018

openshift-ci-robot commented Oct 10, 2018

mfojtik commented Oct 10, 2018

mfojtik Oct 10, 2018

deads2k commented Oct 17, 2018

mfojtik commented Oct 17, 2018

mfojtik commented Oct 17, 2018

abhinavdahiya commented Oct 17, 2018

abhinavdahiya commented Oct 17, 2018

abhinavdahiya commented Oct 17, 2018

wking commented Oct 17, 2018 •

edited

Loading

deads2k commented Oct 17, 2018

deads2k commented Oct 17, 2018

openshift-bot commented Oct 18, 2018

sttts commented Oct 18, 2018

openshift-ci-robot commented Oct 18, 2018

sttts Oct 18, 2018

sttts Oct 18, 2018

mfojtik Oct 18, 2018

sttts Oct 18, 2018

replace kube-core rendering with openshift operators render #420

replace kube-core rendering with openshift operators render #420

Conversation

mfojtik commented Oct 5, 2018

mfojtik commented Oct 5, 2018

mfojtik commented Oct 5, 2018

mfojtik Oct 8, 2018

Choose a reason for hiding this comment

abhinavdahiya commented Oct 8, 2018

derekwaynecarr commented Oct 10, 2018

openshift-ci-robot commented Oct 10, 2018

mfojtik commented Oct 10, 2018

mfojtik Oct 10, 2018

Choose a reason for hiding this comment

deads2k commented Oct 17, 2018

mfojtik commented Oct 17, 2018

mfojtik commented Oct 17, 2018

abhinavdahiya commented Oct 17, 2018

abhinavdahiya commented Oct 17, 2018

abhinavdahiya commented Oct 17, 2018

wking commented Oct 17, 2018 • edited Loading

deads2k commented Oct 17, 2018

deads2k commented Oct 17, 2018

openshift-bot commented Oct 18, 2018

sttts commented Oct 18, 2018

openshift-ci-robot commented Oct 18, 2018

sttts Oct 18, 2018

Choose a reason for hiding this comment

sttts Oct 18, 2018

Choose a reason for hiding this comment

mfojtik Oct 18, 2018

Choose a reason for hiding this comment

sttts Oct 18, 2018

Choose a reason for hiding this comment

wking commented Oct 17, 2018 •

edited

Loading