아래쪽에 CI/CD, Monitoring, Logging에 대한 것은 완료

wordpress - mysql은 미완성. eks cluster 구성과 addon 설치에 대한 것만 기재하였음 (업데이트 예정)

---

Create eks cluster by using ClusterConfig file

github setup directory

ekscreate.yaml

metadata

• name : spcluster
• region : ap-northeast-2 (SEOUL)

Availability Zone

• ap-northeast-2a
• ap-northeast-2b
• ap-northeast-2c
• ap-northeast-2d

iam (For controllers running in the kube-system namespace)

• aws-load-balancer-controller
• ebs-csi-controller-sa
• cluster-autoscaler

managed node group

• instance type : t3.medium
• min size : 2
• desired capacity : 3
• max size : 4
• iam
  • addon policy
    • autoscaler
    • albingress
    • cloudwatch

fargate profile

• name : spfargate
• selector : namespace = spspace

cloudwatch

• cluster logging
  • api
  • audit
  • authenticator
  • controller manager
  • scheduler

Creating cluster command

# create cluster
$ eksctl create cluster -f ekscreate.yaml

# approve iam service account
$ eksctl create iamserviceaccount -f ekscreate.yaml  --approve

# When an warning occurs
$ eksctl create iamserviceaccount -f ekscreate.yaml  --approve --override-existing-serviceaccounts

Install add-ons

aws-load-balancer-contoller

Refer to the docs

# apply target group binding custom resource
$ kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"

# add eks-charts repository
$ helm repo add eks https://aws.github.io/eks-charts

# update local repo
$ helm repo update

# install aws load balancer contoller
# why image tag v2.1.3 = latest version has bug, not working
$ helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller \
  --set clusterName=spcluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set image.tag=v2.1.3 \
  -n kube-system

# verify
$ kubectl get deployment -n kube-system aws-load-balancer-controller

Could not find webhook error while using load balancer

# Since we don't use webhooks, we can solve it by deleting webhook configuration.
$ kubectl delete validatingwebhookconfiguration aws-load-balancer-webhook

EBS CSI Driver

Refer to the docs

# add repo
$ helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver

# repo update
$ helm repo update

# install aws ebs csi driver
$ helm upgrade -install aws-ebs-csi-driver aws-ebs-csi-driver/aws-ebs-csi-driver \
  --namespace kube-system \
  --set enableVolumeResizing=true \
  --set enableVolumeSnapshot=true \
  --set serviceAccount.controller.create=false \
  --set serviceAccount.controller.name=ebs-csi-controller-sa

Metrics Server

Refer to the docs

# Deploy the metrics server
$ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# verify
$ kubectl get deployment metrics-server -n kube-system

Cluster Autoscaler

Refer to the docs

# get cluster-autoscaler-autodiscover.yaml file
$ curl -o cluster-autoscaler-autodiscover.yaml https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml

edit cluster-autoscaler-autodiscover.yaml

Deployment.spec.template.spec.containers.command last line : <YOUR CLUSTER NAME> -> spcluster

# deploy
$ kubectl create -f cluster-autoscaler-autodiscover.yaml

# verify
$ kubectl -n kube-system logs -f deployment.apps/cluster-autoscaler

CloudWatch Container Insights

Refer to the docs

# set variables
$ ClusterName=spcluster
$ RegionName=ap-northeast-2
$ FluentBitHttpPort='2020'
$ FluentBitReadFromHead='Off'
$ [[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
$ [[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'

# deploy
$ curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/;s/{{region_name}}/'${RegionName}'/;s/{{http_server_toggle}}/"'${FluentBitHttpServer}'"/;s/{{http_server_port}}/"'${FluentBitHttpPort}'"/;s/{{read_from_head}}/"'${FluentBitReadFromHead}'"/;s/{{read_from_tail}}/"'${FluentBitReadFromTail}'"/' | kubectl apply -f - 

# verify
$ kubectl get po -n amazon-cloudwatch

---

CI - Github, Dockerhub automated build

Running nginx in ubuntu environment

github root directory

Link github and dockerhub

If you don't want to connect to an account, skip this step
You can connect to the repository only by 'configure automated builds' menu

1. Log in to dockerhub
2. Go to the account settings screen as shown in the figure below.

3. Select github connect from the 'linked account' menu
4. If connected properly, the following screen will be displayed.

Create Dockerfile

$ cat Dockerfile
FROM ubuntu:focal
RUN apt-get update
RUN apt-get install -y nginx
WORKDIR /etc/nginx
CMD ["nginx","-g","daemon off;"]
EXPOSE 80

Create git repository

Upload Dockerfile to github

$ git init
$ git add .
$ git commit -m "init, upload dockfile"
$ git remote add origin <git repositoy address>
$ git branch -M main
$ git push

Create dockerhub repository and set automated build

1. Create repository
2. Go to the build menu of the created repository
3. Select the connected github and select organizaion, repository

   If you skipped the account linking step, select 'configure automated builds' menu

4. Complete the detailed settings created below and save.

5. After checking the status, if additional build is needed, press the trigger.

6. Check if the build was successful.

$ docker run -itd -p 8080:80 [username]/[repository name]

CD - ArgoCD

github deploy directory

For this step, the eks cluster environment must be configured first.

It can be configured by referring to 'Create eks cluster by using clusterconfig file' at the top of this README.

Create deploy files in deploy directory

# Collect the files to be deployed in the deploy folder
$ cd deploy

# Manifest for distributing the built dockerfile
$ cat nginx.yaml
apiVersion: apps/v1
kind: Deployment   
metadata:
  name: nginx
spec:
  replicas: 3
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: dmsqlczzz/nginx-ubuntu:latest
        name: nginx
        ports:
        - containerPort: 80

# Manifest for service        
$ cat svc.yaml 
apiVersion: v1
kind: Service
metadata:
  name: nginxsvc
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx

Upload deploy directory to github

$ git init
$ git add .
$ git commit -m "upload deploy directory"
$ git remote add origin <git repositoy address>
$ git branch -M main
$ git push

Set for the ArgoCD environment

# Create 'argocd' namespace
$ kubectl create namespace argocd

# Install ArgoCD
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Download Argocd CD CLI
$ choco install argocd

# Access the ArgoCD API server
$ kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

# Find ArcoCD IP address
# argocd service resource - External IP
$ kubectl get all -n argocd

# Get password
$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Create ArgoCD app

At chrome secret mode

1. Access to ArgoCD IP address

2. Login ArgoCD
   ID : admin
   PW : <Password found with argocd secret>

3. Create new app

• GENERAL
  • Applicatioin Name : Set as you like
  • Project : default

• SOURCE
  • Repository URL : your github repository url
    You can find it here.
    
  • Path : <Directory where you put the deploy file in github repository>
    In my case, deploy
• DESTINATION
  • Cluster URL : <cluster with argocd>
    If you click this area, you can see the clusters that can be selected.
    In my case, https://kubernetes.default.svc
  • Namespace : default

Click CREATE botton

Sync ArgoCD app

1. Click SYNC button
2. Click SYNCHRONIZE button

If it is well configured, it becomes 'Healthy' and 'Synced' as shown below.

If you click on the area shown above, you can check the currently deployed resource configuration as shown below.

Modify the deploy file and check if argocd works well

1. Modify deploy file

# Modify deploy file
$ cat nginx.yaml
apiVersion: apps/v1
kind: Deployment   
metadata:
  name: nginx
spec:
  replicas: 4 # Modified from 3 to 4
  revisionHistoryLimit: 4 # Modified from 3 to 4
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: dmsqlczzz/nginx-ubuntu:latest
        name: nginx
        ports:
        - containerPort: 80

2. Make git commit and push

$ git add .
$ git commit -m "Modify deploy file"
$ git push

3. Check if argocd works well

As shown below, check that the nginx pods have changed from 3 to 4.

Monitoring - AWS CloudWatch ContainerInsight

When configuring the EKS cluster, CloudWatch Container Insights and Metrics Server are deployed, so monitoring is possible in the AWS console.

Logging - AWS CloudWatch Log

Like monitoring, you can check the logs in AWS CloudWatch.
Filtering the log group by cluster name and checking the log group of my cluster is as follows.

If you click the log group, you can check the log stream as follows.

If you click the log stream, you can see the log details and also check the log details.