This repository has been archived by the owner on Dec 11, 2019. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 4
Kubernetes Dev Cluster
luglo edited this page Nov 22, 2018
·
3 revisions
based on https://github.com/kubernetes/kops/blob/master/docs/aws.md
Install AWS CLI (https://aws.amazon.com/cli/) & configure with credentials
Export AWS Credentials so kops can use them to interact with AWS
export AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id)
export AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key)
Install kubectl (https://kubernetes.io/docs/tasks/tools/install-kubectl/)
Install kops (https://github.com/kubernetes/kops/blob/master/docs/install.md)
Set up IAM Role & User for KOPS to use. Note: Didn't do this, should we? If so replace AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY exports!
Set up S3 Bucket to save cluster state. I had to do this, kops doesnt work without the bucket.
aws s3api create-bucket --bucket fog-chatbot-dev-kcluster-state-store --create-bucket-configuration LocationConstraint=eu-west-1 --region eu-west-1
{
"Location": "http://fog-chatbot-dev-kcluster-state-store.s3.amazonaws.com/"
}
Enable versioning in S3 bucket
aws s3api put-bucket-versioning --bucket fog-chatbot-dev-kcluster-state-store --versioning-configuration Status=Enabled
Save some regularly used strings to environment
export NAME=fog-chatbot-dev.k8s.local
export NODE_SIZE=${NODE_SIZE:-t2.micro}
export MASTER_SIZE=${MASTER_SIZE:-t2.micro}
export ZONES=${ZONES:-"eu-west-1a"}
export KOPS_STATE_STORE=s3://fog-chatbot-dev-kcluster-state-store
Initial create of the cluster ( will no fail because cluster config exists in the bucket)
kops create cluster $NAME \
--node-count 1 \
--zones $ZONES \
--node-size $NODE_SIZE \
--master-size $MASTER_SIZE \
--master-zones $ZONES \
--ssh-public-key ~/.ssh/id_rsa.pub
Cluster is now created
Edit it with (config is downlaoded from s3 and submitted on close):
kops edit cluster ${NAME}
use export EDITOR="subl --wait" to change your default editor
Note: T3 instances are currently not supported by kops
kops get instancegroups --name $NAME
NAME ROLE MACHINETYPE MIN MAX ZONES
master-eu-west-1a Master t3.nano 1 1 eu-west-1a
nodes Node t3.nano 1 1 eu-west-1a
kops edit instancegroup master-eu-west-1a --name $NAME
kops edit instancegroup nodes --name $NAME
This will show the applied changes to aws services
kops update cluster $NAME
Write build script in terraform
kops update cluster $NAME --target=terraform
Deploy current configuration kops update cluster $NAME --yes
kops update cluster $NAME ✔ 20:53:28
I1122 20:54:13.000193 26604 apply_cluster.go:505] Gossip DNS: skipping DNS validation
I1122 20:54:13.022810 26604 executor.go:103] Tasks: 0 done / 72 total; 30 can run
I1122 20:54:15.443421 26604 executor.go:103] Tasks: 30 done / 72 total; 24 can run
I1122 20:54:18.568466 26604 executor.go:103] Tasks: 54 done / 72 total; 16 can run
I1122 20:54:19.171379 26604 executor.go:103] Tasks: 70 done / 72 total; 2 can run
I1122 20:54:19.291152 26604 executor.go:103] Tasks: 72 done / 72 total; 0 can run
Will create resources:
AutoscalingGroup/master-eu-west-1a.masters.fog-chatbot-dev.k8s.local
MinSize 1
MaxSize 1
Subnets [name:eu-west-1a.fog-chatbot-dev.k8s.local]
Tags {KubernetesCluster: fog-chatbot-dev.k8s.local, k8s.io/role/master: 1, Name: master-eu-west-1a.masters.fog-chatbot-dev.k8s.local}
Granularity 1Minute
Metrics [GroupDesiredCapacity, GroupInServiceInstances, GroupMaxSize, GroupMinSize, GroupPendingInstances, GroupStandbyInstances, GroupTerminatingInstances, GroupTotalInstances]
LaunchConfiguration name:master-eu-west-1a.masters.fog-chatbot-dev.k8s.local
AutoscalingGroup/nodes.fog-chatbot-dev.k8s.local
MinSize 1
MaxSize 1
Subnets [name:eu-west-1a.fog-chatbot-dev.k8s.local]
Tags {k8s.io/role/node: 1, Name: nodes.fog-chatbot-dev.k8s.local, KubernetesCluster: fog-chatbot-dev.k8s.local}
Granularity 1Minute
Metrics [GroupDesiredCapacity, GroupInServiceInstances, GroupMaxSize, GroupMinSize, GroupPendingInstances, GroupStandbyInstances, GroupTerminatingInstances, GroupTotalInstances]
LaunchConfiguration name:nodes.fog-chatbot-dev.k8s.local
DHCPOptions/fog-chatbot-dev.k8s.local
DomainName eu-west-1.compute.internal
DomainNameServers AmazonProvidedDNS
Shared false
Tags {KubernetesCluster: fog-chatbot-dev.k8s.local, kubernetes.io/cluster/fog-chatbot-dev.k8s.local: owned, Name: fog-chatbot-dev.k8s.local}
EBSVolume/a.etcd-events.fog-chatbot-dev.k8s.local
AvailabilityZone eu-west-1a
VolumeType gp2
SizeGB 20
Encrypted false
Tags {k8s.io/etcd/events: a/a, k8s.io/role/master: 1, kubernetes.io/cluster/fog-chatbot-dev.k8s.local: owned, Name: a.etcd-events.fog-chatbot-dev.k8s.local, KubernetesCluster: fog-chatbot-dev.k8s.local}
EBSVolume/a.etcd-main.fog-chatbot-dev.k8s.local
AvailabilityZone eu-west-1a
VolumeType gp2
SizeGB 20
Encrypted false
Tags {Name: a.etcd-main.fog-chatbot-dev.k8s.local, KubernetesCluster: fog-chatbot-dev.k8s.local, k8s.io/etcd/main: a/a, k8s.io/role/master: 1, kubernetes.io/cluster/fog-chatbot-dev.k8s.local: owned}
IAMInstanceProfile/masters.fog-chatbot-dev.k8s.local
Shared false
IAMInstanceProfile/nodes.fog-chatbot-dev.k8s.local
Shared false
IAMInstanceProfileRole/masters.fog-chatbot-dev.k8s.local
InstanceProfile name:masters.fog-chatbot-dev.k8s.local id:masters.fog-chatbot-dev.k8s.local
Role name:masters.fog-chatbot-dev.k8s.local
IAMInstanceProfileRole/nodes.fog-chatbot-dev.k8s.local
InstanceProfile name:nodes.fog-chatbot-dev.k8s.local id:nodes.fog-chatbot-dev.k8s.local
Role name:nodes.fog-chatbot-dev.k8s.local
IAMRole/masters.fog-chatbot-dev.k8s.local
ExportWithID masters
IAMRole/nodes.fog-chatbot-dev.k8s.local
ExportWithID nodes
IAMRolePolicy/masters.fog-chatbot-dev.k8s.local
Role name:masters.fog-chatbot-dev.k8s.local
IAMRolePolicy/nodes.fog-chatbot-dev.k8s.local
Role name:nodes.fog-chatbot-dev.k8s.local
InternetGateway/fog-chatbot-dev.k8s.local
VPC name:fog-chatbot-dev.k8s.local
Shared false
Tags {kubernetes.io/cluster/fog-chatbot-dev.k8s.local: owned, Name: fog-chatbot-dev.k8s.local, KubernetesCluster: fog-chatbot-dev.k8s.local}
LaunchConfiguration/master-eu-west-1a.masters.fog-chatbot-dev.k8s.local
ImageID kope.io/k8s-1.6-debian-jessie-amd64-hvm-ebs-2018-08-17
InstanceType t2.micro
SSHKey name:kubernetes.fog-chatbot-dev.k8s.local-67:fd:90:30:1f:c3:9b:5e:01:ed:4a:bc:f1:d1:f0:2c id:kubernetes.fog-chatbot-dev.k8s.local-67:fd:90:30:1f:c3:9b:5e:01:ed:4a:bc:f1:d1:f0:2c
SecurityGroups [name:masters.fog-chatbot-dev.k8s.local]
AssociatePublicIP true
IAMInstanceProfile name:masters.fog-chatbot-dev.k8s.local id:masters.fog-chatbot-dev.k8s.local
RootVolumeSize 64
RootVolumeType gp2
SpotPrice
LaunchConfiguration/nodes.fog-chatbot-dev.k8s.local
ImageID kope.io/k8s-1.6-debian-jessie-amd64-hvm-ebs-2018-08-17
InstanceType t2.micro
SSHKey name:kubernetes.fog-chatbot-dev.k8s.local-67:fd:90:30:1f:c3:9b:5e:01:ed:4a:bc:f1:d1:f0:2c id:kubernetes.fog-chatbot-dev.k8s.local-67:fd:90:30:1f:c3:9b:5e:01:ed:4a:bc:f1:d1:f0:2c
SecurityGroups [name:nodes.fog-chatbot-dev.k8s.local]
AssociatePublicIP true
IAMInstanceProfile name:nodes.fog-chatbot-dev.k8s.local id:nodes.fog-chatbot-dev.k8s.local
RootVolumeSize 128
RootVolumeType gp2
SpotPrice
Route/0.0.0.0/0
RouteTable name:fog-chatbot-dev.k8s.local
CIDR 0.0.0.0/0
InternetGateway name:fog-chatbot-dev.k8s.local
RouteTable/fog-chatbot-dev.k8s.local
VPC name:fog-chatbot-dev.k8s.local
Shared false
Tags {Name: fog-chatbot-dev.k8s.local, KubernetesCluster: fog-chatbot-dev.k8s.local, kubernetes.io/cluster/fog-chatbot-dev.k8s.local: owned, kubernetes.io/kops/role: public}
RouteTableAssociation/eu-west-1a.fog-chatbot-dev.k8s.local
RouteTable name:fog-chatbot-dev.k8s.local
Subnet name:eu-west-1a.fog-chatbot-dev.k8s.local
SSHKey/kubernetes.fog-chatbot-dev.k8s.local-67:fd:90:30:1f:c3:9b:5e:01:ed:4a:bc:f1:d1:f0:2c
KeyFingerprint d6:75:95:82:97:54:ee:67:84:33:3a:12:0d:a2:5a:3f
SecurityGroup/masters.fog-chatbot-dev.k8s.local
Description Security group for masters
VPC name:fog-chatbot-dev.k8s.local
RemoveExtraRules [port=22, port=443, port=2380, port=2381, port=4001, port=4002, port=4789, port=179]
Tags {KubernetesCluster: fog-chatbot-dev.k8s.local, kubernetes.io/cluster/fog-chatbot-dev.k8s.local: owned, Name: masters.fog-chatbot-dev.k8s.local}
SecurityGroup/nodes.fog-chatbot-dev.k8s.local
Description Security group for nodes
VPC name:fog-chatbot-dev.k8s.local
RemoveExtraRules [port=22]
Tags {Name: nodes.fog-chatbot-dev.k8s.local, KubernetesCluster: fog-chatbot-dev.k8s.local, kubernetes.io/cluster/fog-chatbot-dev.k8s.local: owned}
SecurityGroupRule/all-master-to-master
SecurityGroup name:masters.fog-chatbot-dev.k8s.local
SourceGroup name:masters.fog-chatbot-dev.k8s.local
SecurityGroupRule/all-master-to-node
SecurityGroup name:nodes.fog-chatbot-dev.k8s.local
SourceGroup name:masters.fog-chatbot-dev.k8s.local
SecurityGroupRule/all-node-to-node
SecurityGroup name:nodes.fog-chatbot-dev.k8s.local
SourceGroup name:nodes.fog-chatbot-dev.k8s.local
SecurityGroupRule/https-external-to-master-0.0.0.0/0
SecurityGroup name:masters.fog-chatbot-dev.k8s.local
CIDR 0.0.0.0/0
Protocol tcp
FromPort 443
ToPort 443
SecurityGroupRule/master-egress
SecurityGroup name:masters.fog-chatbot-dev.k8s.local
CIDR 0.0.0.0/0
Egress true
SecurityGroupRule/node-egress
SecurityGroup name:nodes.fog-chatbot-dev.k8s.local
CIDR 0.0.0.0/0
Egress true
SecurityGroupRule/node-to-master-tcp-1-2379
SecurityGroup name:masters.fog-chatbot-dev.k8s.local
Protocol tcp
FromPort 1
ToPort 2379
SourceGroup name:nodes.fog-chatbot-dev.k8s.local
SecurityGroupRule/node-to-master-tcp-2382-4000
SecurityGroup name:masters.fog-chatbot-dev.k8s.local
Protocol tcp
FromPort 2382
ToPort 4000
SourceGroup name:nodes.fog-chatbot-dev.k8s.local
SecurityGroupRule/node-to-master-tcp-4003-65535
SecurityGroup name:masters.fog-chatbot-dev.k8s.local
Protocol tcp
FromPort 4003
ToPort 65535
SourceGroup name:nodes.fog-chatbot-dev.k8s.local
SecurityGroupRule/node-to-master-udp-1-65535
SecurityGroup name:masters.fog-chatbot-dev.k8s.local
Protocol udp
FromPort 1
ToPort 65535
SourceGroup name:nodes.fog-chatbot-dev.k8s.local
SecurityGroupRule/ssh-external-to-master-0.0.0.0/0
SecurityGroup name:masters.fog-chatbot-dev.k8s.local
CIDR 0.0.0.0/0
Protocol tcp
FromPort 22
ToPort 22
SecurityGroupRule/ssh-external-to-node-0.0.0.0/0
SecurityGroup name:nodes.fog-chatbot-dev.k8s.local
CIDR 0.0.0.0/0
Protocol tcp
FromPort 22
ToPort 22
Subnet/eu-west-1a.fog-chatbot-dev.k8s.local
ShortName eu-west-1a
VPC name:fog-chatbot-dev.k8s.local
AvailabilityZone eu-west-1a
CIDR 172.20.32.0/19
Shared false
Tags {SubnetType: Public, Name: eu-west-1a.fog-chatbot-dev.k8s.local, KubernetesCluster: fog-chatbot-dev.k8s.local, kubernetes.io/cluster/fog-chatbot-dev.k8s.local: owned, kubernetes.io/role/elb: 1}
VPC/fog-chatbot-dev.k8s.local
CIDR 172.20.0.0/16
EnableDNSHostnames true
EnableDNSSupport true
Shared false
Tags {KubernetesCluster: fog-chatbot-dev.k8s.local, kubernetes.io/cluster/fog-chatbot-dev.k8s.local: owned, Name: fog-chatbot-dev.k8s.local}
VPCDHCPOptionsAssociation/fog-chatbot-dev.k8s.local
VPC name:fog-chatbot-dev.k8s.local
DHCPOptions name:fog-chatbot-dev.k8s.local
Must specify --yes to apply changes
kops update cluster $NAME --yes ✔ 20:54:19
I1122 20:55:02.082169 26668 apply_cluster.go:505] Gossip DNS: skipping DNS validation
I1122 20:55:02.931592 26668 executor.go:103] Tasks: 0 done / 72 total; 30 can run
I1122 20:55:08.334715 26668 executor.go:103] Tasks: 30 done / 72 total; 24 can run
I1122 20:55:12.314177 26668 executor.go:103] Tasks: 54 done / 72 total; 16 can run
I1122 20:55:15.199967 26668 launchconfiguration.go:380] waiting for IAM instance profile "nodes.fog-chatbot-dev.k8s.local" to be ready
I1122 20:55:26.140586 26668 executor.go:103] Tasks: 70 done / 72 total; 2 can run
I1122 20:55:27.010929 26668 executor.go:103] Tasks: 72 done / 72 total; 0 can run
I1122 20:55:27.513397 26668 update_cluster.go:290] Exporting kubecfg for cluster
W1122 20:55:27.990422 26668 create_kubecfg.go:75] Did not find API endpoint for gossip hostname; may not be able to reach cluster
kops has set your kubectl context to fog-chatbot-dev.k8s.local
Cluster changes have been applied to the cloud.
Changes may require instances to restart: kops rolling-update cluster
After that change the server value in .kube/config to the public dns of the ec2 master instance.
e.g. server: https://ec2-34-242-145-96.eu-west-1.compute.amazonaws.com
Now you can run kop / kubctl commands to inspect/modify/destroy your kubernetes cluster. (--insecure-skip-tls-verify is necessary becaue the automaticlally created certificates are only valid for the not existing k8s.local hostname. this is an open issue in kops https://github.com/kubernetes/kops/issues/2990)
kubectl cluster-info --insecure-skip-tls-verify
kops delete cluster --name ${NAME} 1 ↵ 21:39:11
TYPE NAME ID
autoscaling-config master-eu-west-1a.masters.fog-chatbot-dev.k8s.local-20181122195514 master-eu-west-1a.masters.fog-chatbot-dev.k8s.local-20181122195514
autoscaling-config nodes.fog-chatbot-dev.k8s.local-20181122195514 nodes.fog-chatbot-dev.k8s.local-20181122195514
autoscaling-group master-eu-west-1a.masters.fog-chatbot-dev.k8s.local master-eu-west-1a.masters.fog-chatbot-dev.k8s.local
autoscaling-group nodes.fog-chatbot-dev.k8s.local nodes.fog-chatbot-dev.k8s.local
dhcp-options fog-chatbot-dev.k8s.local dopt-0a6c97cbe79e8356a
iam-instance-profile masters.fog-chatbot-dev.k8s.local masters.fog-chatbot-dev.k8s.local
iam-instance-profile nodes.fog-chatbot-dev.k8s.local nodes.fog-chatbot-dev.k8s.local
iam-role masters.fog-chatbot-dev.k8s.local masters.fog-chatbot-dev.k8s.local
iam-role nodes.fog-chatbot-dev.k8s.local nodes.fog-chatbot-dev.k8s.local
instance master-eu-west-1a.masters.fog-chatbot-dev.k8s.local i-03f92c721cbeb82c9
instance nodes.fog-chatbot-dev.k8s.local i-033ff4c681c22a119
internet-gateway fog-chatbot-dev.k8s.local igw-0f5e3f272fd1b770d
keypair kubernetes.fog-chatbot-dev.k8s.local-67:fd:90:30:1f:c3:9b:5e:01:ed:4a:bc:f1:d1:f0:2c kubernetes.fog-chatbot-dev.k8s.local-67:fd:90:30:1f:c3:9b:5e:01:ed:4a:bc:f1:d1:f0:2c
route-table fog-chatbot-dev.k8s.local rtb-0a2c8e8289fd63b34
security-group masters.fog-chatbot-dev.k8s.local sg-0ffb6e3c243a91e57
security-group nodes.fog-chatbot-dev.k8s.local sg-050f5062d2ad5d383
subnet eu-west-1a.fog-chatbot-dev.k8s.local subnet-0843b54061d287448
volume a.etcd-events.fog-chatbot-dev.k8s.local vol-075d87e019a6564a4
volume a.etcd-main.fog-chatbot-dev.k8s.local vol-0af609afb91715381
vpc fog-chatbot-dev.k8s.local vpc-0f6bc67eaa35f71dd
Must specify --yes to delete cluster
Final delete
kops delete cluster --name ${NAME} --yes