This project enables you to provision the infrastructure and demo web application required for the AWS EKS Cluster Blue/Green Upgrade lab. This demo is designed to help DevOps engineers who want to upgrade their AWS EKS cluster with a blue/green strategy.
This demo is based on the ap-northeast-2. (Seoul region)
- You'll need your own domain with name-servers configurable.
- This demo uses resources not included in the free-tier, so starting the demo may cost some money.
- In this demo, 3 new VPCs are created. Therefore, it is necessary to check whether the VPC service has enough quota in advance to avoid being limited by service quotas.
- The three new VPCs each use the CIDR address ranges of
172.51.0.0/16,172.61.0.0/16, and10.100.0.0/24. If the VPC for the current production workload overlaps with the corresponding CIDR address range, some code must be modified to prevent the demo environment from affecting the production workload.- How to modify a CIDR address
- To modify the CIDR address range for an EKS cluster, see my-eks-blueprints.ts.
- To modify the CIDR address range for an Request Client, see request-client-stack.ts.
- To modify the CIDR address range for an EKS cluster, see my-eks-blueprints.ts.
- How to modify a CIDR address
- You should check the subdomain used in this demo to prevent the demo environment from impacting the operating workload. In the demo, it is basically programmed to receive requests by using a subdomain named 'weighted' for this purpose. (e.g.,
weighted.example.com) If you want to use a different subdomain for the demo, you'll need to modify some code.
- You'll need your own domain with name-servers configurable.
- AWS Account
- Create an AWS account and set the AdministratorAccess permission to a user in that account.
- The user's permissions are required to provision AWS resources such as VPCs, EKS, and ALBs required for the demo.
- AWS CLI
- Install the AWS CLI, and set up the aws credentials on the PC where you want to perform the demo.
- AWS CDK
- docker
- For a smooth demo, the lab equipment must have the Docker Daemon running.
- If Docker Daemon is not running, the following errors may occur
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
- npm
- kubectl
- git
1-1. Clone that project to your local PC with the git clone command.
git clone [email protected]:aws-samples/eks-cluster-upgrade-with-a-blue-green-strategy.git1-2. To utilize your own domain in the demo, define environment variables in the terminal as shown in the command below.
export CDK_HOSTED_ZONE_NAME=<domain>Use the following command to check if the environment variable has been defined correctly.
echo $CDK_HOSTED_ZONE_NAME1-3. If a public hosted zone for the domain is already configured in Route53 and you want to use that public hosted zone for demo, define the environment variables as shown in the following command.
export CDK_IS_HOSTED_ZONE_ALREADY_EXISTS=trueIf Route53 does not have a public hosted zone configured for the domain, define an environment variable as shown in the following command to create a new public hosted zone.
export CDK_IS_HOSTED_ZONE_ALREADY_EXISTS=falseUse the following command to check if the environment variable has been defined correctly.
echo $CDK_IS_HOSTED_ZONE_ALREADY_EXISTSIf you don't set an environment variable, a new public hosted zone will be created by default.
If a public hosted zone does not exist but the CDK_IS_HOSTED_ZONE_ALREADY_EXISTS environment variable is set to true, an error will occur.
1-4. In this demo, we’ll generate user traffic using the open source load testing tool Locust. As we modify weighted traffic from the blue cluster to the green cluster on Route53, we’ll continuously monitor the EKS Ingress requests using CloudWatch.
Locust requires a username and password for their dashboard GUI. The admin account information needed to sign-in to the Request Client dashboard is defined as environment variables as shown below.
export CDK_REQUEST_CLIENT_USERNAME=<username>
export CDK_REQUEST_CLIENT_PASSWORD=<password>Use the following command to check if the environment variable has been defined correctly.
echo $CDK_REQUEST_CLIENT_USERNAME
echo $CDK_REQUEST_CLIENT_PASSWORDIf you do not set account information separately, the username is set to awsuser and the password is set to passw0rd by default.
cd aws-cdks/my-eks-blueprints/
npm install
cdk bootstrap
cdk synth
cdk deploy --allPlease check for each stack provisioned through the terminal and type y to the question.
Through this task, provision the architecture shown in the figure above. It takes approximately 15-30 minutes.
Once the entire CDK Stack is provisioned, refer to the Public hosted zone created on Route53 to change the domain nameserver settings.
3-1. From the Route 53 Console, select Public hosted zone. Next, check the Name servers information for the selected public hosted zone.
3-2. Next, go to the site where you purchased the domain and update the name server information.
How to update a domain's name servers must be checked on the site where the domain was purchased.
Access the dashboard of the Network Load Balancer (Locust) through a browser. The endpoint you need to connect to is the endpoint of the Application Load Balancer, which is created when provisioning the request-client Stack.
Access the dashboard with the account information you set up in step 1.
If you log in and access the dashboard, you can see the following screen.
Next, enter the domain you specified in Host.
When entering Host, add http:// and weighted subdomains.
Below is an example :
http://weighted.my-domain.com
The reason for added a subdomain called weighted to the domain is to prevent the demo environment from affecting production workloads.
In the deploy-demo-application.ts, you can see the code that changes the value of the external-dns.alpha.kubernetes.io/hostname annotation in the ingress manifest to the domain with the weighted subdomain added.
Since most production workloads don't use weighted as a subdomain, this demo used weighted subdomain.
If you want to specify a separate subdomain, refer to the weightedDomain variable in the deploy-demo-application.ts code and modify it to the desired value.
Next, click the Start swarming button to start generating traffic.
If traffic is generated normally, you can check the graph in the Charts tab as shown below.
The Route53 weighted routing feature adjusts the weight value of traffic flowing to the blue and green clusters. As shown in the example below, you can adjust the weight values to gradually shift traffic from the blue cluster to the green cluster.
- 1st :
BLUE(80%):GREEN(20%) - 2nd :
BLUE(50%):GREEN(50%) - 3rd :
BLUE(20%):GREEN(80%) - finally :
BLUE(0%):GREEN(100%)
Note that the initial weighting values are set to BLUE(100%):GREEN(0%).
The weight value can be adjusted by modifying the value of the external-dns.alpha.kubernetes.io/aws-weight annotation within demo-aplication-blue.yaml and demo-application-green.yaml. The value is entered as a percentage.
After modifying the weight values in the code, redeploy the CDK Stack to update the weighted routing settings for the Route53 record.
cd aws-cdks/my-eks-blueprints/
cdk deploy --allFor rollbacks, you can do the opposite and update the weight values to BLUE(100%):GREEN(0%).
Go to the CloudWatch console and refer to the CloudWatch dashboard named EKS_Monitoring_Dashboard to monitor the process of changing the weight value to make sure it works properly.
cd aws-cdks/my-eks-blueprints/
cdk destroy --allSee CONTRIBUTING for more information.
This library is licensed under the MIT-0 License. See the LICENSE file.