diff --git a/README.md b/README.md index 1ac6be788b..68caea4cfb 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ One needs to run the `imagebuilder` command instead of the `docker build`. Note: this info is RH only, it needs to be backported every time the `README.md` is synced with the upstream one. -## How to deploy and test the machine controller with minikube +## Deploy machine API plane with minikube 1. **Install kvm** @@ -38,79 +38,169 @@ Note: this info is RH only, it needs to be backported every time the `README.md` 2. **Deploying the cluster** - Because of [cluster-api#475](https://github.com/kubernetes-sigs/cluster-api/issues/475) the minikube version can't be higher than `0.28.0`. - To install minikube `v0.28.0`, you can run: + To install minikube `v1.1.0`, you can run: ```sg - $ curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.28.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/ + $ curl -Lo minikube https://storage.googleapis.com/minikube/releases/v1.1.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/ ``` To deploy the cluster: ``` - minikube start --vm-driver kvm2 - eval $(minikube docker-env) + $ minikube start --vm-driver kvm2 --kubernetes-version v1.13.1 --v 5 + $ eval $(minikube docker-env) ``` -3. **Building the machine controller** +3. **Deploying machine API controllers** + For development purposes the aws machine controller itself will run out of the machine API stack. + Otherwise, docker images needs to be built, pushed into a docker registry and deployed within the stack. + + To deploy the stack: ``` - $ make -C cmd/machine-controller + kustomize build config | kubectl apply -f - ``` -4. **Deploying the cluster-api stack manifests** +4. **Deploy secret with AWS credentials** - Add your AWS credentials to the `addons.yaml` file (in base64 - format). You can either do this manually or use the - `examples/render-aws-secrets.sh`. + AWS actuator assumes existence of a secret file (references in machine object) with base64 encoded credentials: - The easy deployment is: + ```yaml + apiVersion: v1 + kind: Secret + metadata: + name: aws-credentials-secret + namespace: default + type: Opaque + data: + aws_access_key_id: FILLIN + aws_secret_access_key: FILLIN + ``` - ```sh - ./examples/render-aws-secrets.sh examples/addons.yaml | kubectl apply -f - - ``` + You can use `examples/render-aws-secrets.sh` script to generate the secret: + ```sh + ./examples/render-aws-secrets.sh examples/addons.yaml | kubectl apply -f - + ``` - The manual deployment is: +5. **Provision AWS resource** - ``` sh - $ echo -n 'your_id' | base64 - $ echo -n 'your_key' | base64 - $ kubectl apply -f examples/addons.yaml - ``` + The actuator expects existence of certain resource in AWS such as: + - vpc + - subnets + - security groups + - etc. - Deploy CRDs: + To create them, you can run: - ```sh - $ kubectl apply -f config/crd/machine.crd.yaml - $ kubectl apply -f config/crd/machineset.crd.yaml - $ kubectl apply -f config/crd/machinedeployment.crd.yaml - $ kubectl apply -f config/crd/cluster.crd.yaml - ``` + ```sh + $ ENVIRONMENT_ID=aws-actuator-k8s ./hack/aws-provision.sh install + ``` - Deploy machine API controllers: + To delete the resources, you can run: - ```sh - $ kubectl apply -f config/rbac/rbac_role.yaml - $ kubectl apply -f config/rbac/rbac_role_binding.yaml - $ kubectl apply -f config/controllers/deployment.yaml - ``` + ```sh + $ ENVIRONMENT_ID=aws-actuator-k8s ./hack/aws-provision.sh destroy + ``` - Deploy the cluster manifest: - ```sh - $ kubectl apply -f examples/cluster.yaml - ``` + All machine manifests expect `ENVIRONMENT_ID` to be set to `aws-actuator-k8s`. - Deploy the machines: +## Test locally built aws actuator - ```sh - $ kubectl apply -f examples/machine.yaml --validate=false - ``` +1. **Tear down machine-controller** - or alternatively: + Deployed machine API plane (`machine-api-controllers` deployment) is (among other + controllers) running `machine-controller`. In order to run locally built one, + simply edit `machine-api-controllers` deployment and remove `machine-controller` container from it. - ```sh - $ kubectl apply -f examples/machine-set.yaml --validate=false - ``` +1. **Build and run aws actuator outside of the cluster** + + ```sh + $ go build -o bin/manager sigs.k8s.io/cluster-api-provider-aws/cmd/manager + ``` + + ```sh + $ ./bin/manager --kubeconfig ~/.kube/config --logtostderr -v 5 -alsologtostderr + ``` + +1. **Deploy k8s apiserver through machine manifest**: + + To deploy user data secret with kubernetes apiserver initialization (under [config/master-user-data-secret.yaml](config/master-user-data-secret.yaml)): + + ```yaml + $ kubectl apply -f config/master-user-data-secret.yaml + ``` + + To deploy kubernetes master machine (under [config/master-machine.yaml](config/master-machine.yaml)): + + ```yaml + $ kubectl apply -f config/master-machine.yaml + ``` + +1. **Pull kubeconfig from created master machine** + + The master public IP can be accessed from AWS Portal. Once done, you + can collect the kube config by running: + + ``` + $ ssh -i SSHPMKEY ec2-user@PUBLICIP 'sudo cat /root/.kube/config' > kubeconfig + $ kubectl --kubeconfig=kubeconfig config set-cluster kubernetes --server=https://PUBLICIP:8443 + ``` + + Once done, you can access the cluster via `kubectl`. E.g. + + ```sh + $ kubectl --kubeconfig=kubeconfig get nodes + ``` + +## Deploy k8s cluster in AWS with machine API plane deployed + +1. **Generate bootstrap user data** + + To generate bootstrap script for machine api plane, simply run: + + ```sh + $ ./config/generate-bootstrap.sh + ``` + + The script requires `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables to be set. + It generates `config/bootstrap.yaml` secret for master machine + under `config/master-machine.yaml`. + + The generated bootstrap secret contains user data responsible for: + - deployment of kube-apiserver + - deployment of machine API plane with aws machine controllers + - generating worker machine user data script secret deploying a node + - deployment of worker machineset + +1. **Deploy machine API plane through machine manifest**: + + First, deploy generated bootstrap secret: + + ```yaml + $ kubectl apply -f config/bootstrap.yaml + ``` + + Then, deploy master machine (under [config/master-machine.yaml](config/master-machine.yaml)): + + ```yaml + $ kubectl apply -f config/master-machine.yaml + ``` + +1. **Pull kubeconfig from created master machine** + + The master public IP can be accessed from AWS Portal. Once done, you + can collect the kube config by running: + + ``` + $ ssh -i SSHPMKEY ec2-user@PUBLICIP 'sudo cat /root/.kube/config' > kubeconfig + $ kubectl --kubeconfig=kubeconfig config set-cluster kubernetes --server=https://PUBLICIP:8443 + ``` + + Once done, you can access the cluster via `kubectl`. E.g. + + ```sh + $ kubectl --kubeconfig=kubeconfig get nodes + ``` # Upstream Implementation Other branches of this repository may choose to track the upstream diff --git a/config/bootstrap.sh b/config/bootstrap.sh new file mode 100644 index 0000000000..119690467f --- /dev/null +++ b/config/bootstrap.sh @@ -0,0 +1,155 @@ +#!/bin/bash + +cat < /root/user-data.sh +#!/bin/bash + +################################################ +######## Install packages and binaries +################################################ + +cat < /etc/yum.repos.d/kubernetes.repo +[kubernetes] +name=Kubernetes +baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 +enabled=1 +gpgcheck=1 +repo_gpgcheck=1 +gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg +exclude=kube* +EOF + +setenforce 0 +yum install -y docker +systemctl enable docker +systemctl start docker +yum install -y kubelet-1.13.1 kubeadm-1.13.1 kubectl-1.13.1 kubernetes-cni-0.6.0-0 --disableexcludes=kubernetes + +cat < /etc/default/kubelet +KUBELET_KUBEADM_EXTRA_ARGS=--cgroup-driver=systemd +EOF + +echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables + +curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases/latest |\ + grep browser_download |\ + grep linux |\ + cut -d '"' -f 4 |\ + xargs curl -O -L +chmod u+x kustomize_*_linux_amd64 +sudo mv kustomize_*_linux_amd64 /usr/bin/kustomize + +sudo yum install -y git + +################################################ +######## Deploy kubernetes master +################################################ + +kubeadm init --apiserver-bind-port 8443 --token 2iqzqm.85bs0x6miyx1nm7l --apiserver-cert-extra-sans=$(curl icanhazip.com) --pod-network-cidr=192.168.0.0/16 -v 6 + +# Enable networking by default. +kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml --kubeconfig /etc/kubernetes/admin.conf + +# Binaries expected under /opt/cni/bin are actually under /usr/libexec/cni +if [[ ! -e /opt/cni/bin ]]; then + mkdir -p /opt/cni/bin + cp /usr/libexec/cni/bridge /opt/cni/bin + cp /usr/libexec/cni/loopback /opt/cni/bin + cp /usr/libexec/cni/host-local /opt/cni/bin +fi + +mkdir -p /root/.kube +cp -i /etc/kubernetes/admin.conf /root/.kube/config +chown $(id -u):$(id -g) /root/.kube/config + +################################################ +######## Deploy machine-api plane +################################################ + +git clone https://github.com/ingvagabund/cluster-api-provider-aws.git +cd cluster-api-provider-aws +git checkout k8s-bootstrap + +cat < secret.yaml +apiVersion: v1 +kind: Secret +metadata: + name: aws-credentials-secret + namespace: default +type: Opaque +data: + aws_access_key_id: FILLIN + aws_secret_access_key: FILLIN +EOF + +sudo kubectl apply -f secret.yaml + +kustomize build config | sudo kubectl apply -f - + +kubectl apply -f config/master-user-data-secret.yaml +kubectl apply -f config/master-machine.yaml + +################################################ +######## generate worker machineset user data +################################################ + +cat < /root/workerset-user-data.sh +#!/bin/bash + +cat < /root/workerset-user-data.sh +#!/bin/bash + +cat < /etc/yum.repos.d/kubernetes.repo +[kubernetes] +name=Kubernetes +baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 +enabled=1 +gpgcheck=1 +repo_gpgcheck=1 +gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg +exclude=kube* +EOF +setenforce 0 +yum install -y docker +systemctl enable docker +systemctl start docker +yum install -y kubelet-1.13.1 kubeadm-1.13.1 kubernetes-cni-0.6.0-0 --disableexcludes=kubernetes + +cat < /etc/default/kubelet +KUBELET_KUBEADM_EXTRA_ARGS=--cgroup-driver=systemd +EOF + +echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables + +kubeadm join $(curl icanhazip.com):8443 --token 2iqzqm.85bs0x6miyx1nm7l --discovery-token-unsafe-skip-ca-verification +WORKERHEREDOC + +bash /root/workerset-user-data.sh 2>&1 > /root/workerset-user-data.logs + +WORKERSET + +################################################ +######## deploy worker user data and machineset +################################################ + +# NOTE: The secret is rendered twice, the first time when it's run during bootstrapping. +# During bootstrapping, /root/workerset-user-data.sh does not exist yet. +# So \$ needs to be used so the command is executed the second time +# the script is executed. + +cat < /root/worker-secret.yaml +apiVersion: v1 +kind: Secret +metadata: + name: worker-user-data-secret + namespace: default +type: Opaque +data: + userData: \$(cat /root/workerset-user-data.sh | base64 --w=0) +EOF + +sudo kubectl apply -f /root/worker-secret.yaml + +sudo kubectl apply -f config/worker-machineset.yaml +HEREDOC + +bash /root/user-data.sh 2>&1 > /root/user-data.logs diff --git a/config/controllers/deployment.yaml b/config/controllers/deployment.yaml index 109e92e33d..e82d94889a 100644 --- a/config/controllers/deployment.yaml +++ b/config/controllers/deployment.yaml @@ -1,16 +1,13 @@ apiVersion: apps/v1 kind: Deployment metadata: - name: clusterapi-manager-controllers - namespace: test + name: machine-api-controllers + namespace: default labels: api: clusterapi k8s-app: controller tectonic-operators.coreos.com/managed-by: machine-api-operator spec: - securityContext: - runAsNonRoot: true - runAsUser: 65534 selector: matchLabels: api: clusterapi @@ -22,6 +19,9 @@ spec: api: clusterapi k8s-app: controller spec: + securityContext: + runAsNonRoot: true + runAsUser: 65534 nodeSelector: node-role.kubernetes.io/master: "" tolerations: diff --git a/config/crds/cluster.crd.yaml b/config/crds/cluster.crd.yaml deleted file mode 100644 index a66738eef5..0000000000 --- a/config/crds/cluster.crd.yaml +++ /dev/null @@ -1,92 +0,0 @@ -apiVersion: apiextensions.k8s.io/v1beta1 -kind: CustomResourceDefinition -metadata: - creationTimestamp: null - labels: - controller-tools.k8s.io: "1.0" - name: clusters.cluster.k8s.io -spec: - group: cluster.k8s.io - names: - kind: Cluster - plural: clusters - scope: Namespaced - subresources: - status: {} - validation: - openAPIV3Schema: - properties: - apiVersion: - type: string - kind: - type: string - metadata: - type: object - spec: - properties: - clusterNetwork: - properties: - pods: - properties: - cidrBlocks: - items: - type: string - type: array - required: - - cidrBlocks - type: object - serviceDomain: - type: string - services: - properties: - cidrBlocks: - items: - type: string - type: array - required: - - cidrBlocks - type: object - required: - - services - - pods - - serviceDomain - type: object - providerSpec: - properties: - value: - type: object - valueFrom: - type: object - type: object - required: - - clusterNetwork - type: object - status: - properties: - apiEndpoints: - items: - properties: - host: - type: string - port: - format: int64 - type: integer - required: - - host - - port - type: object - type: array - errorMessage: - type: string - errorReason: - type: string - providerStatus: - type: object - type: object - version: v1alpha1 -status: - acceptedNames: - kind: "" - plural: "" - conditions: [] - storedVersions: [] diff --git a/config/crds/machine.crd.yaml b/config/crds/machine.openshift.io.crd.yaml similarity index 65% rename from config/crds/machine.crd.yaml rename to config/crds/machine.openshift.io.crd.yaml index 0907f0c0ef..54fd350cd1 100644 --- a/config/crds/machine.crd.yaml +++ b/config/crds/machine.openshift.io.crd.yaml @@ -1,12 +1,35 @@ apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: - creationTimestamp: null labels: controller-tools.k8s.io: "1.0" - name: machines.cluster.k8s.io + name: machines.machine.openshift.io spec: - group: cluster.k8s.io + additionalPrinterColumns: + - JSONPath: .status.providerStatus.instanceId + name: Instance + description: Instance ID of machine created in AWS + type: string + - JSONPath: .status.providerStatus.instanceState + name: State + description: State of the AWS instance + type: string + - JSONPath: .spec.providerSpec.value.instanceType + name: Type + description: Type of instance + type: string + - JSONPath: .spec.providerSpec.value.placement.region + name: Region + description: Region associated with machine + type: string + - JSONPath: .spec.providerSpec.value.placement.availabilityZone + name: Zone + description: Zone associated with machine + type: string + - JSONPath: .metadata.creationTimestamp + name: Age + type: date + group: machine.openshift.io names: kind: Machine plural: machines @@ -23,6 +46,7 @@ spec: metadata: type: object spec: + type: object properties: configSource: type: object @@ -33,6 +57,12 @@ spec: value: type: object valueFrom: + properties: + machineClass: + properties: + provider: + type: string + type: object type: object type: object taints: @@ -50,7 +80,6 @@ spec: type: object required: - providerSpec - type: object status: properties: addresses: @@ -82,7 +111,7 @@ spec: - kubelet type: object type: object - version: v1alpha1 + version: v1beta1 status: acceptedNames: kind: "" diff --git a/config/crds/machinedeployment.crd.yaml b/config/crds/machinedeployment.openshift.io.crd.yaml similarity index 84% rename from config/crds/machinedeployment.crd.yaml rename to config/crds/machinedeployment.openshift.io.crd.yaml index 9f26e75f8f..6f54dbec23 100644 --- a/config/crds/machinedeployment.crd.yaml +++ b/config/crds/machinedeployment.openshift.io.crd.yaml @@ -1,17 +1,20 @@ apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: - creationTimestamp: null labels: controller-tools.k8s.io: "1.0" - name: machinedeployments.cluster.k8s.io + name: machinedeployments.machine.openshift.io spec: - group: cluster.k8s.io + group: machine.openshift.io names: kind: MachineDeployment plural: machinedeployments scope: Namespaced subresources: + scale: + labelSelectorPath: .status.labelSelector + specReplicasPath: .spec.replicas + statusReplicasPath: .status.replicas status: {} validation: openAPIV3Schema: @@ -23,6 +26,7 @@ spec: metadata: type: object spec: + type: object properties: minReadySeconds: format: int32 @@ -44,19 +48,19 @@ spec: properties: rollingUpdate: properties: - maxSurge: - type: object - maxUnavailable: - type: object + maxSurge: {} + maxUnavailable: {} type: object type: type: string type: object template: + type: object properties: metadata: type: object spec: + type: object properties: configSource: type: object @@ -67,6 +71,12 @@ spec: value: type: object valueFrom: + properties: + machineClass: + properties: + provider: + type: string + type: object type: object type: object taints: @@ -84,12 +94,9 @@ spec: type: object required: - providerSpec - type: object - type: object required: - selector - template - type: object status: properties: availableReplicas: @@ -111,7 +118,7 @@ spec: format: int32 type: integer type: object - version: v1alpha1 + version: v1beta1 status: acceptedNames: kind: "" diff --git a/config/crds/machineset.crd.yaml b/config/crds/machineset.openshift.io.crd.yaml similarity index 82% rename from config/crds/machineset.crd.yaml rename to config/crds/machineset.openshift.io.crd.yaml index 09f5fc6608..ff12050c03 100644 --- a/config/crds/machineset.crd.yaml +++ b/config/crds/machineset.openshift.io.crd.yaml @@ -1,17 +1,20 @@ apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: - creationTimestamp: null labels: controller-tools.k8s.io: "1.0" - name: machinesets.cluster.k8s.io + name: machinesets.machine.openshift.io spec: - group: cluster.k8s.io + group: machine.openshift.io names: kind: MachineSet plural: machinesets scope: Namespaced subresources: + scale: + labelSelectorPath: .status.labelSelector + specReplicasPath: .spec.replicas + statusReplicasPath: .status.replicas status: {} validation: openAPIV3Schema: @@ -23,6 +26,7 @@ spec: metadata: type: object spec: + type: object properties: minReadySeconds: format: int32 @@ -33,10 +37,12 @@ spec: selector: type: object template: + type: object properties: metadata: type: object spec: + type: object properties: configSource: type: object @@ -47,6 +53,12 @@ spec: value: type: object valueFrom: + properties: + machineClass: + properties: + provider: + type: string + type: object type: object type: object taints: @@ -64,11 +76,8 @@ spec: type: object required: - providerSpec - type: object - type: object required: - selector - type: object status: properties: availableReplicas: @@ -93,7 +102,7 @@ spec: required: - replicas type: object - version: v1alpha1 + version: v1beta1 additionalPrinterColumns: - JSONPath: .spec.replicas description: Desired Replicas @@ -107,6 +116,10 @@ spec: description: Ready Replicas name: Ready type: integer + - JSONPath: .status.availableReplicas + name: Available + description: Observed number of available replicas + type: string - JSONPath: .metadata.creationTimestamp name: Age type: date diff --git a/config/generate-bootstrap.sh b/config/generate-bootstrap.sh new file mode 100755 index 0000000000..eb885407a7 --- /dev/null +++ b/config/generate-bootstrap.sh @@ -0,0 +1,29 @@ +#!/bin/sh + +if [ -z "$AWS_ACCESS_KEY_ID" ]; then + echo "error: AWS_ACCESS_KEY_ID is not set in the environment" 2>&1 + exit 1 +fi + +if [ -z "$AWS_SECRET_ACCESS_KEY" ]; then + echo "error: AWS_SECRET_ACCESS_KEY is not set in the environment" 2>&1 + exit 1 +fi + +script_dir="$(cd $(dirname "${BASH_SOURCE[0]}") && pwd -P)" + +secrethash=$(cat $script_dir/bootstrap.sh | \ + sed "s/ aws_access_key_id: FILLIN/ aws_access_key_id: $(echo -n $AWS_ACCESS_KEY_ID | base64)/" | \ + sed "s/ aws_secret_access_key: FILLIN/ aws_secret_access_key: $(echo -n $AWS_SECRET_ACCESS_KEY | base64)/" | \ + base64 --w=0) + +cat < $script_dir/bootstrap.yaml +apiVersion: v1 +kind: Secret +metadata: + name: master-user-data-secret + namespace: default +type: Opaque +data: + userData: $secrethash +EOF diff --git a/config/default/kustomization.yaml b/config/kustomization.yaml similarity index 66% rename from config/default/kustomization.yaml rename to config/kustomization.yaml index 6d8ee203b1..1031f1955b 100644 --- a/config/default/kustomization.yaml +++ b/config/kustomization.yaml @@ -1,12 +1,12 @@ # Adds namespace to all resources. -namespace: migration-system +# namespace: default # Value of this field is prepended to the # names of all resources, e.g. a deployment named # "wordpress" becomes "alices-wordpress". # Note that it should also match with the prefix (text before '-') of the namespace # field above. -namePrefix: migration- +#namePrefix: migration- # Labels to add to all resources and selectors. #commonLabels: @@ -18,13 +18,9 @@ namePrefix: migration- # YAML string, with resources separated by document # markers ("---"). resources: -- ../crds/machine.crd.yaml -- ../crds/machineset.crd.yaml -- ../crds/machinedeployment.crd.yaml -- ../crds/cluster.crd.yaml -- ../rbac/rbac_role.yaml -- ../rbac/rbac_role_binding.yaml -- ../controllers/deployment.yaml - -#patches: -#- manager_image_patch.yaml +- crds/machine.openshift.io.crd.yaml +- crds/machineset.openshift.io.crd.yaml +- crds/machinedeployment.openshift.io.crd.yaml +- rbac/rbac_role.yaml +- rbac/rbac_role_binding.yaml +- controllers/deployment.yaml diff --git a/examples/master-machine.yaml b/config/master-machine.yaml similarity index 56% rename from examples/master-machine.yaml rename to config/master-machine.yaml index 2c91e3a7a5..46cb754841 100644 --- a/examples/master-machine.yaml +++ b/config/master-machine.yaml @@ -1,20 +1,18 @@ --- -apiVersion: "cluster.k8s.io/v1alpha1" +apiVersion: machine.openshift.io/v1beta1 kind: Machine metadata: - name: aws-actuator-testing-machine-master - namespace: test - generateName: vs-master- + name: master-machine + namespace: default labels: - machine.openshift.io/cluster-api-cluster: tb-asg-35 + machine.openshift.io/cluster-api-cluster: aws-actuator-k8s spec: metadata: labels: node-role.kubernetes.io/master: "" - node-role.kubernetes.io/infra: "" providerSpec: value: - apiVersion: awsproviderconfig.k8s.io/v1alpha1 + apiVersion: awsproviderconfig.openshift.io/v1beta1 kind: AWSMachineProviderConfig ami: filters: @@ -35,25 +33,17 @@ spec: filters: - name: "tag:Name" values: - - "{{ .ClusterID }}-worker-*" + - "aws-actuator-k8s-worker-*" iamInstanceProfile: id: openshift_master_launch_instances keyName: libra - tags: - - name: openshift-node-group-config - value: node-config-master - - name: host-type - value: master - - name: sub-host-type - value: default securityGroups: - filters: - name: "tag:Name" values: - - "{{ .ClusterID }}-*" + - "aws-actuator-k8s-*" publicIp: true userDataSecret: - name: aws-actuator-master-user-data-secret - versions: - kubelet: 1.10.1 - controlPlane: 1.10.1 + name: master-user-data-secret + credentialsSecret: + name: aws-credentials-secret diff --git a/examples/master-userdata.yaml b/config/master-user-data-secret.yaml similarity index 65% rename from examples/master-userdata.yaml rename to config/master-user-data-secret.yaml index d032904a97..2d72bfcbdc 100644 --- a/examples/master-userdata.yaml +++ b/config/master-user-data-secret.yaml @@ -1,8 +1,8 @@ apiVersion: v1 kind: Secret metadata: - name: aws-actuator-master-user-data-secret - namespace: test + name: master-user-data-secret + namespace: default type: Opaque data: userData: | @@ -13,8 +13,8 @@ data: MQpyZXBvX2dwZ2NoZWNrPTEKZ3Bna2V5PWh0dHBzOi8vcGFja2FnZXMuY2xvdWQuZ29vZ2xlLmNv bS95dW0vZG9jL3l1bS1rZXkuZ3BnIGh0dHBzOi8vcGFja2FnZXMuY2xvdWQuZ29vZ2xlLmNvbS95 dW0vZG9jL3JwbS1wYWNrYWdlLWtleS5ncGcKZXhjbHVkZT1rdWJlKgpFT0YKc2V0ZW5mb3JjZSAw - Cnl1bSBpbnN0YWxsIC15IGt1YmVsZXQtMS4xMS4zIGt1YmVhZG0tMS4xMS4zIGt1YmVjdGwtMS4x - MS4zIC0tZGlzYWJsZWV4Y2x1ZGVzPWt1YmVybmV0ZXMKCmNhdCA8PEVPRiA+IC9ldGMvZGVmYXVs + Cnl1bSBpbnN0YWxsIC15IGt1YmVsZXQtMS4xMy4xIGt1YmVhZG0tMS4xMy4xIGt1YmVjdGwtMS4x + My4xIC0tZGlzYWJsZWV4Y2x1ZGVzPWt1YmVybmV0ZXMKCmNhdCA8PEVPRiA+IC9ldGMvZGVmYXVs dC9rdWJlbGV0CktVQkVMRVRfS1VCRUFETV9FWFRSQV9BUkdTPS0tY2dyb3VwLWRyaXZlcj1zeXN0 ZW1kCkVPRgoKZWNobyAnMScgPiAvcHJvYy9zeXMvbmV0L2JyaWRnZS9icmlkZ2UtbmYtY2FsbC1p cHRhYmxlcwoKa3ViZWFkbSBpbml0IC0tYXBpc2VydmVyLWJpbmQtcG9ydCA4NDQzIC0tdG9rZW4g @@ -25,7 +25,12 @@ data: ZHI9MTkyLjE2OC4wLjAvMTYgLXYgNgoKIyBFbmFibGUgbmV0d29ya2luZyBieSBkZWZhdWx0Lgpr dWJlY3RsIGFwcGx5IC1mIGh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9jbG91ZG5h dGl2ZWxhYnMva3ViZS1yb3V0ZXIvbWFzdGVyL2RhZW1vbnNldC9rdWJlYWRtLWt1YmVyb3V0ZXIu - eWFtbCAtLWt1YmVjb25maWcgL2V0Yy9rdWJlcm5ldGVzL2FkbWluLmNvbmYKCm1rZGlyIC1wIC9y - b290Ly5rdWJlCmNwIC1pIC9ldGMva3ViZXJuZXRlcy9hZG1pbi5jb25mIC9yb290Ly5rdWJlL2Nv - bmZpZwpjaG93biAkKGlkIC11KTokKGlkIC1nKSAvcm9vdC8ua3ViZS9jb25maWcKSEVSRURPQwoK - YmFzaCAvcm9vdC91c2VyLWRhdGEuc2ggPiAvcm9vdC91c2VyLWRhdGEubG9ncwo= + eWFtbCAtLWt1YmVjb25maWcgL2V0Yy9rdWJlcm5ldGVzL2FkbWluLmNvbmYKCiMgQmluYXJpZXMg + ZXhwZWN0ZWQgdW5kZXIgL29wdC9jbmkvYmluIGFyZSBhY3R1YWxseSB1bmRlciAvdXNyL2xpYmV4 + ZWMvY25pCmlmIFtbICEgLWUgL29wdC9jbmkvYmluIF1dOyB0aGVuCiAgbWtkaXIgLXAgL29wdC9j + bmkvYmluCiAgY3AgL3Vzci9saWJleGVjL2NuaS9icmlkZ2UgL29wdC9jbmkvYmluCiAgY3AgL3Vz + ci9saWJleGVjL2NuaS9sb29wYmFjayAvb3B0L2NuaS9iaW4KICBjcCAvdXNyL2xpYmV4ZWMvY25p + L2hvc3QtbG9jYWwgL29wdC9jbmkvYmluCmZpCgpta2RpciAtcCAvcm9vdC8ua3ViZQpjcCAtaSAv + ZXRjL2t1YmVybmV0ZXMvYWRtaW4uY29uZiAvcm9vdC8ua3ViZS9jb25maWcKY2hvd24gJChpZCAt + dSk6JChpZCAtZykgL3Jvb3QvLmt1YmUvY29uZmlnCkhFUkVET0MKCmJhc2ggL3Jvb3QvdXNlci1k + YXRhLnNoID4gL3Jvb3QvdXNlci1kYXRhLmxvZ3MK diff --git a/examples/master-user-data.sh b/config/master-user-data.sh similarity index 78% rename from examples/master-user-data.sh rename to config/master-user-data.sh index 30a8d22336..c0b0160af8 100755 --- a/examples/master-user-data.sh +++ b/config/master-user-data.sh @@ -14,7 +14,7 @@ gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cl exclude=kube* EOF setenforce 0 -yum install -y kubelet-1.11.3 kubeadm-1.11.3 kubectl-1.11.3 --disableexcludes=kubernetes +yum install -y kubelet-1.13.1 kubeadm-1.13.1 kubectl-1.13.1 --disableexcludes=kubernetes cat < /etc/default/kubelet KUBELET_KUBEADM_EXTRA_ARGS=--cgroup-driver=systemd @@ -27,6 +27,14 @@ kubeadm init --apiserver-bind-port 8443 --token 2iqzqm.85bs0x6miyx1nm7l --apiser # Enable networking by default. kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml --kubeconfig /etc/kubernetes/admin.conf +# Binaries expected under /opt/cni/bin are actually under /usr/libexec/cni +if [[ ! -e /opt/cni/bin ]]; then + mkdir -p /opt/cni/bin + cp /usr/libexec/cni/bridge /opt/cni/bin + cp /usr/libexec/cni/loopback /opt/cni/bin + cp /usr/libexec/cni/host-local /opt/cni/bin +fi + mkdir -p /root/.kube cp -i /etc/kubernetes/admin.conf /root/.kube/config chown $(id -u):$(id -g) /root/.kube/config diff --git a/config/rbac/rbac_role.yaml b/config/rbac/rbac_role.yaml index 0ec40df288..243861b578 100644 --- a/config/rbac/rbac_role.yaml +++ b/config/rbac/rbac_role.yaml @@ -1,3 +1,4 @@ +--- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: @@ -9,6 +10,17 @@ rules: resources: - clusters - clusters/status + verbs: + - get + - list + - watch + - create + - update + - patch + - delete +- apiGroups: + - machine.openshift.io + resources: - machines - machines/status - machinesets @@ -35,10 +47,84 @@ rules: - update - patch - delete - - apiGroups: - - healthchecking.openshift.io + - "" + resources: + - nodes/status + - pods/status + verbs: + - patch +- apiGroups: + - "" + resources: + - pods/eviction + verbs: + - create +- apiGroups: + - "" + resources: + - pods + verbs: + - get + - watch + - create + - delete + - list +- apiGroups: + - policy + resources: + - poddisruptionbudgets + verbs: + - list + - watch +- apiGroups: + - "" + resources: + - services + - secrets + - replicationcontrollers + - persistentvolumes + - persistentvolumeclaims + verbs: + - list + - watch +- apiGroups: + - apps + resources: + - statefulsets + - replicasets + verbs: + - list + - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - list + - watch +- apiGroups: + - extensions + resources: + - daemonsets + verbs: + - list + - watch + - get +- apiGroups: + - "" resources: - - '*' + - configmaps verbs: - - '*' + - get + - create + - update +- apiGroups: + - "" + resources: + - events + verbs: + - create + - watch + - list + - patch diff --git a/config/rbac/rbac_role_binding.yaml b/config/rbac/rbac_role_binding.yaml index 52ac059065..ceac526d86 100644 --- a/config/rbac/rbac_role_binding.yaml +++ b/config/rbac/rbac_role_binding.yaml @@ -9,4 +9,4 @@ roleRef: subjects: - kind: ServiceAccount name: default - namespace: test + namespace: default diff --git a/examples/worker-machineset.yaml b/config/worker-machineset.yaml similarity index 63% rename from examples/worker-machineset.yaml rename to config/worker-machineset.yaml index 1a962b778b..4e6b95484a 100644 --- a/examples/worker-machineset.yaml +++ b/config/worker-machineset.yaml @@ -1,29 +1,29 @@ --- -apiVersion: cluster.k8s.io/v1alpha1 +apiVersion: machine.openshift.io/v1beta1 kind: MachineSet metadata: - name: default-worker-machineset - namespace: test + name: worker-machineset + namespace: default labels: - machine.openshift.io/cluster-api-cluster: tb-asg-35 + machine.openshift.io/cluster-api-cluster: aws-actuator-k8s spec: replicas: 1 selector: matchLabels: machine.openshift.io/cluster-api-machineset: worker-machine - machine.openshift.io/cluster-api-cluster: tb-asg-35 + machine.openshift.io/cluster-api-cluster: aws-actuator-k8s template: metadata: labels: machine.openshift.io/cluster-api-machineset: worker-machine - machine.openshift.io/cluster-api-cluster: tb-asg-35 + machine.openshift.io/cluster-api-cluster: aws-actuator-k8s spec: metadata: labels: node-role.kubernetes.io/compute: "" providerSpec: value: - apiVersion: awsproviderconfig.k8s.io/v1alpha1 + apiVersion: awsproviderconfig.openshift.io/v1beta1 kind: AWSMachineProviderConfig ami: filters: @@ -46,25 +46,17 @@ spec: filters: - name: "tag:Name" values: - - "{{ .ClusterID }}-worker-*" + - "aws-actuator-k8s-worker-*" iamInstanceProfile: id: openshift_master_launch_instances keyName: libra - tags: - - name: openshift-node-group-config - value: node-config-worker - - name: host-type - value: worker - - name: sub-host-type - value: default securityGroups: - filters: - name: "tag:Name" values: - - "{{ .ClusterID }}-*" + - "aws-actuator-k8s-*" publicIp: true userDataSecret: - name: aws-actuator-node-user-data-secret - versions: - kubelet: 1.10.1 - controlPlane: 1.10.1 + name: worker-user-data-secret + credentialsSecret: + name: aws-credentials-secret diff --git a/examples/addons.yaml b/examples/addons.yaml index 59a93edce1..c2e1c4f722 100644 --- a/examples/addons.yaml +++ b/examples/addons.yaml @@ -1,14 +1,9 @@ apiVersion: v1 -kind: Namespace -metadata: - name: test ---- -apiVersion: v1 kind: Secret metadata: name: aws-credentials-secret - namespace: test + namespace: default type: Opaque data: - awsAccessKeyId: - awsSecretAccessKey: + aws_access_key_id: + aws_secret_access_key: diff --git a/examples/render-aws-secrets.sh b/examples/render-aws-secrets.sh index d84cb1ce0d..d500adc156 100755 --- a/examples/render-aws-secrets.sh +++ b/examples/render-aws-secrets.sh @@ -20,6 +20,6 @@ fi x=$(echo -n "$AWS_ACCESS_KEY_ID" | base64) y=$(echo -n "$AWS_SECRET_ACCESS_KEY" | base64) -sed -e "s/awsAccessKeyId:.*/awsAccessKeyId: $x/" \ - -e "s/awsSecretAccessKey:.*/awsSecretAccessKey: $y/" \ +sed -e "s/aws_access_key_id:.*/aws_access_key_id: $x/" \ + -e "s/aws_secret_access_key:.*/aws_secret_access_key: $y/" \ "$1"