Start rendering assets using cluster-kube-apiserver operator renderer #322

mfojtik · 2018-09-25T17:01:02Z

This PR will start generating assets using the new cluster-kube-apiserver-operator image.
Along with manifests and bootstrap static pods that should be used in future by bootkube start, it will provide secrets and configmaps we can feed the openshift-api-operator.

[mfojtik@dev-centos generated]$ docker run -v $(pwd)/tls:/assets --rm openshift/origin-cluster-kube-apiserver-operator:latest /usr/bin/cluster-kube-apiserver-operator render --asset-input-dir=/assets --asset-output-dir=/assets/kube-apiserver-bootstrap --config-output-file=/assets/kube-apiserver-bootstrap/config --config-override-file=/usr/share/bootkube/manifests/config/config-overrides.yaml
Writing asset: /assets/kube-apiserver-bootstrap/bootstrap-manifests/kube-apiserver-pod.yaml
Writing asset: /assets/kube-apiserver-bootstrap/manifests/etcd-service.yaml
Writing asset: /assets/kube-apiserver-bootstrap/manifests/openshift-kube-apiserver-ns.yaml
Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-aggregator-client.yaml
Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-etcd-client.yaml
Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-serving-cert.yaml
Writing asset: /assets/kube-apiserver-bootstrap/manifests/configmap-client-ca.yaml
Writing asset: /assets/kube-apiserver-bootstrap/manifests/configmap-kubelet-serving-ca.yaml
Writing asset: /assets/kube-apiserver-bootstrap/manifests/configmap-sa-token-signing-certs.yaml
Writing asset: /assets/kube-apiserver-bootstrap/manifests/kube-apiserver-daemonset.yaml
Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-kubelet-client.yaml
Writing asset: /assets/kube-apiserver-bootstrap/manifests/configmap-aggregator-client-ca.yaml
Writing asset: /assets/kube-apiserver-bootstrap/manifests/configmap-etcd-serving-ca.yaml
Writing asset: /assets/kube-apiserver-bootstrap/manifests/configmap-kube-apiserver-config.yaml

/cc @deads2k
/cc @juanvallejo
/cc @sttts

mfojtik · 2018-09-25T17:01:34Z

modules/bootkube/resources/bootkube.sh

+
+	# shellcheck disable=SC2154
+	podman run \
+		--volume "$PWD:/assets:z" \


this assumes the $PWD has the generated/tls secrets, is that assumption correct?

this assumes the $PWD has the generated/tls secrets, is that assumption correct?

The working directory is set here, so the generated TLS will be in ${PWD}/tls. So no generated directory, but I think you're handling this correctly.

yeah, ${PWD}/tls sounds what I want, thanks!

mfojtik · 2018-10-01T11:19:29Z

@smarterclayton @deads2k I updated this to fit into new go templates and also added the controller manager render. Once we are confident that the our operators provide same experience as kube-core control plane, we can switch it over just by copying the manifests we need.

Also this demonstrates that our operator rendering functionality works (we probably should think about the image versions in bootkube.go as currently the :latest is used.

deads2k · 2018-10-01T12:32:58Z

@abhinavdahiya this is needed to set up the resources used by our operators. Can you have a look?

/assign @abhinavdahiya

abhinavdahiya · 2018-10-01T16:50:29Z

This PR is more useful

if we drop the corresponding things from tectonic-operators kube-core-operator bump it here
use the new rendered assets for bootstrapping.

Otherwise this is unused code-path.
cc @crawford

deads2k · 2018-10-01T17:11:57Z

This PR is more useful

if we drop the corresponding things from tectonic-operators kube-core-operator bump it here

use the new rendered assets for bootstrapping.

Otherwise this is unused code-path.
cc @crawford

We create additional/different resources. Doesn't this start producing those? We want to enable new things and see them work before removing old.

abhinavdahiya · 2018-10-01T18:47:08Z

We create additional/different resources. Doesn't this start producing those? We want to enable new things and see them work before removing old.

I meant to say these new files are rendered on disk but not actually used.

deads2k · 2018-10-01T19:29:25Z

I meant to say these new files are rendered on disk but not actually used.

Where do we put them to have them created?

abhinavdahiya · 2018-10-01T20:45:30Z

@deads2k

installer/pkg/asset/ignition/content/bootkube.go

Lines 32 to 47 in 7f8f397

    
           if [ ! -d kco-bootstrap ] 
        
           then 
        
           	echo "Rendering Kubernetes core manifests..." 
        
           	# shellcheck disable=SC2154 
        
           	podman run \ 
        
           		--volume "$PWD:/assets:z" \ 
        
           		--volume /etc/kubernetes:/etc/kubernetes:z \ 
        
           		"{{.KubeCoreRenderImage}}" \ 
        
           		--config=/assets/kco-config.yaml \ 
        
           		--output=/assets/kco-bootstrap 
        
           	cp --recursive kco-bootstrap/bootstrap-configs /etc/kubernetes/bootstrap-configs 
        
           	cp --recursive kco-bootstrap/bootstrap-manifests . 
        
           	cp --recursive kco-bootstrap/manifests . 
        
           fi

kube-core-operator renders it assets in 3 dirs:

$ ls -l /opt/tectonic/kco-bootstrap/
bootstrap-configs
bootstrap-manifests
manifests

bootstrap-configs
This is copied to /etc/kubernetes/bootstrap-configs. And is used by bootstrap control plane.
bootstrap-manifests
This is copied to /opt/tectonic; this is then used by bootkube start as source of bootstrap control plane static pods.
manifests
This merged using cp with already existing /opt/tectonic/manifets dir. This /opt/tectonic/manifets is used by bootkube start to push manifests in cluster when api is up.
/opt/tectonic/tls and /opt/tectonic/auth
These directories have the tls assets and kubeconfig for bootstrap control plane respectively. These are also used by bootkube start.

https://github.com/kubernetes-incubator/bootkube/blob/master/pkg/bootkube/bootstrap.go#L28

abhinavdahiya · 2018-10-03T16:52:54Z

@deads2k @mfojtik any progress on this?

You can now get image for your operator using https://github.com/openshift/installer/blob/master/pkg/asset/ignition/content/bootkube.go#L32

mfojtik · 2018-10-03T17:24:58Z

@abhinavdahiya updated, i think this can merge (even if it is a no-op for now) and we can figure out what manifest to copy where as second step.

/cc @smarterclayton
/cc @deads2k

pkg/asset/ignition/content/bootkube.go

abhinavdahiya · 2018-10-03T17:31:02Z

/approve

abhinavdahiya · 2018-10-03T19:35:34Z

pkg/asset/ignition/content/bootkube.go

+		--config-output-file=/assets/kube-controller-manager-bootstrap/config
+
+	# TODO: copy the bootstrap manifests to replace kube-core-operator
+	cp --recursive kube-apiserver-bootstrap/manifests/openshift-kube-controller-manager-ns.yaml manifests/00_openshift-kube-controller-manager-ns.yaml


cp --recursive kube-controller-manager-bootstrap/manifests/openshift-kube-controller-manager-ns.yaml manifests/00_openshift-kube-controller-manager-ns.yaml

mfojtik · 2018-10-03T19:36:11Z

still having bootkube fail the first time with

bootkube.sh[808]: cp: cannot stat ‘kube-apiserver-bootstrap/manifests/openshift-kube-controller-manager-ns.yaml’: No such file or directory

@sjenning fixed, I hate bash...

@abhinavdahiya can you re-tag please, hopefully last time...

abhinavdahiya · 2018-10-03T19:36:16Z

/lgtm

openshift-ci-robot · 2018-10-03T19:36:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, mfojtik

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [abhinavdahiya]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

abhinavdahiya · 2018-10-03T19:49:56Z

/retest

mfojtik · 2018-10-03T20:20:59Z

/retest

openshift-bot · 2018-10-03T20:21:21Z

/retest

Please review the full test history for this PR and help us cut down flakes.

wking · 2018-10-03T20:28:19Z

With multiple folks (including the bot ;) banging away on /retest, it's helpful (for me at least) to drop few lines from the error you saw into a comment here (e.g. here). That makes it easier to see if we're hitting the same error each time, in which case it's likely to be a real bug and not a temporary flake.

wking · 2018-10-03T21:09:50Z

e2e:

error: .status.conditions accessor error: Failure is of the type string, expected map[string]interface{}
timeout waiting for router to be available
2018/10/03 20:56:49 Container test in pod e2e-aws failed, exit code 1, reason Error
2018

We've seen that before, e.g. here. It's a wait flake.

/retest

openshift-bot · 2018-10-04T00:23:21Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2018-10-04T02:24:19Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2018-10-04T04:24:18Z

/retest

Please review the full test history for this PR and help us cut down flakes.

mfojtik · 2018-10-04T08:25:02Z

smoke:

could not wait for pod to complete: could not wait for pod completion: the pod ci-op-ytkl853l/e2e-aws-smoke failed after 2h3m33s (failed containers: setup, test):  unknown

Container setup exited with code 1, reason Error
Container test exited with code 1, reason Error

Waiting for API at https://ci-op-0gmd4k3x-1d3f3-api.origin-ci-int-aws.dev.rhcloud.com:6443 to respond ...
Waiting for API at https://ci-op-0gmd4k3x-1d3f3-api.origin-ci-int-aws.dev.rhcloud.com:6443 to respond ...
Waiting for API at https://ci-op-0gmd4k3x-1d3f3-api.origin-ci-int-aws.dev.rhcloud.com:6443 to respond ...
Waiting for API at https://ci-op-0gmd4k3x-1d3f3-api.origin-ci-int-aws.dev.rhcloud.com:6443 to respond ...
Waiting for API at https://ci-op-0gmd4k3x-1d3f3-api.origin-ci-int-aws.dev.rhcloud.com:6443 to respond ...
Waiting for API at https://ci-op-0gmd4k3x-1d3f3-api.origin-ci-int-aws.dev.rhcloud.com:6443 to respond ...
Another process exited
2018/10/04 06:28:21 Container test in pod e2e-aws failed, exit code 1, reason Error

i suspect this is this PR fault?

mfojtik · 2018-10-04T08:25:10Z

/retest

openshift-bot · 2018-10-04T08:26:18Z

/retest

Please review the full test history for this PR and help us cut down flakes.

mfojtik · 2018-10-04T11:21:47Z

failure was:

Error: Error applying plan:

3 error(s) occurred:

* module.bootstrap.aws_iam_role.bootstrap: 1 error(s) occurred:

* aws_iam_role.bootstrap: Error creating IAM Role ci-op-0gmd4k3x-1d3f3-bootstrap-role: EntityAlreadyExists: Role with name ci-op-0gmd4k3x-1d3f3-bootstrap-role already exists.
	status code: 409, request id: 64ced276-c7af-11e8-b362-6b3d06569f8f
* module.masters.aws_iam_role.master_role: 1 error(s) occurred:

* aws_iam_role.master_role: Error creating IAM Role ci-op-0gmd4k3x-1d3f3-master-role: EntityAlreadyExists: Role with name ci-op-0gmd4k3x-1d3f3-master-role already exists.
	status code: 409, request id: 64ceab1c-c7af-11e8-9915-c37dbaece9fa
* module.iam.aws_iam_role.worker_role: 1 error(s) occurred:

* aws_iam_role.worker_role: Error creating IAM Role ci-op-0gmd4k3x-1d3f3-worker-role: EntityAlreadyExists: Role with name ci-op-0gmd4k3x-1d3f3-worker-role already exists.
	status code: 409, request id: 64ce5d08-c7af-11e8-b78d-41b07c69b921

/retest

mfojtik · 2018-10-04T11:54:17Z

failed with:

which: no extended.test in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
/bin/bash: line 93: ginkgo: command not found

/retest

openshift-bot · 2018-10-04T12:27:53Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2018-10-04T14:28:51Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2018-10-04T16:30:24Z

/retest

Please review the full test history for this PR and help us cut down flakes.

wking · 2018-10-04T17:37:29Z

/hold

Waiting on #415 to unstick CI.

wking · 2018-10-04T18:36:36Z

#415 is in.

/hold cancel
/retest

openshift-ci-robot · 2018-10-04T20:07:13Z

@mfojtik: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-smoke	74084ed09739dbbe30946e4b892e007f97aed1c9	link	`/test e2e-aws-smoke`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2018-10-04T20:31:52Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot requested review from deads2k, juanvallejo and sttts September 25, 2018 17:01

openshift-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Sep 25, 2018

mfojtik commented Sep 25, 2018

View reviewed changes

mfojtik force-pushed the add-kube-api-server branch 4 times, most recently from db98924 to 74084ed Compare September 26, 2018 18:36

openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 30, 2018

mfojtik force-pushed the add-kube-api-server branch from 74084ed to 6bce6ba Compare October 1, 2018 09:59

openshift-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 1, 2018

openshift-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 1, 2018

openshift-ci-robot assigned abhinavdahiya Oct 1, 2018

abhinavdahiya mentioned this pull request Oct 3, 2018

libvirt: increase master memory to 3GB #408

Merged

mfojtik force-pushed the add-kube-api-server branch from 6bce6ba to 96d533e Compare October 3, 2018 17:23

openshift-ci-robot requested a review from smarterclayton October 3, 2018 17:24

mfojtik force-pushed the add-kube-api-server branch from 96d533e to fc0e120 Compare October 3, 2018 17:26

abhinavdahiya reviewed Oct 3, 2018

View reviewed changes

pkg/asset/ignition/content/bootkube.go Show resolved Hide resolved

abhinavdahiya suggested changes Oct 3, 2018

View reviewed changes

abhinavdahiya approved these changes Oct 3, 2018

View reviewed changes

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 3, 2018

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 4, 2018

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 4, 2018

openshift-merge-robot merged commit 14440c3 into openshift:master Oct 4, 2018

abhinavdahiya mentioned this pull request Oct 4, 2018

Render openshift-cluster-kube-scheduler-operator assets #381

Merged

Start rendering assets using cluster-kube-apiserver operator renderer #322

Start rendering assets using cluster-kube-apiserver operator renderer #322

Conversation

mfojtik commented Sep 25, 2018 • edited Loading

mfojtik Sep 25, 2018

Choose a reason for hiding this comment

wking Sep 25, 2018

Choose a reason for hiding this comment

mfojtik Sep 25, 2018

Choose a reason for hiding this comment

mfojtik commented Oct 1, 2018

deads2k commented Oct 1, 2018

abhinavdahiya commented Oct 1, 2018

deads2k commented Oct 1, 2018

abhinavdahiya commented Oct 1, 2018

deads2k commented Oct 1, 2018

abhinavdahiya commented Oct 1, 2018

abhinavdahiya commented Oct 3, 2018 • edited Loading

mfojtik commented Oct 3, 2018

abhinavdahiya commented Oct 3, 2018

abhinavdahiya Oct 3, 2018

Choose a reason for hiding this comment

mfojtik commented Oct 3, 2018

abhinavdahiya commented Oct 3, 2018

openshift-ci-robot commented Oct 3, 2018

abhinavdahiya commented Oct 3, 2018

mfojtik commented Oct 3, 2018

openshift-bot commented Oct 3, 2018

wking commented Oct 3, 2018

wking commented Oct 3, 2018

openshift-bot commented Oct 4, 2018

openshift-bot commented Oct 4, 2018

openshift-bot commented Oct 4, 2018

mfojtik commented Oct 4, 2018

mfojtik commented Oct 4, 2018

openshift-bot commented Oct 4, 2018

mfojtik commented Oct 4, 2018

mfojtik commented Oct 4, 2018

openshift-bot commented Oct 4, 2018

openshift-bot commented Oct 4, 2018

openshift-bot commented Oct 4, 2018

wking commented Oct 4, 2018

wking commented Oct 4, 2018

openshift-ci-robot commented Oct 4, 2018 • edited Loading

openshift-bot commented Oct 4, 2018

mfojtik commented Sep 25, 2018 •

edited

Loading

abhinavdahiya commented Oct 3, 2018 •

edited

Loading

openshift-ci-robot commented Oct 4, 2018 •

edited

Loading