Moving log collector script to Amazon eks ami repo #243
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nithu0115
changed the title
|
I think this needs some updates to work with v1.4.0 and later @nithu0115. |
|
WIP |
shabir61
added a commit
to OpenGov/amazon-eks-ami
that referenced
this pull request
Apr 20, 2020
* Moving log collector script to Amazon eks ami repo (awslabs#243) * Moving log collector script to this repo * Added changes according to 1.4.1 * Update eks-log-collector.sh URL on readme The instructions on readme were still pointing at the original repository. Updating to reflect the new location * remove kubectl dependency (awslabs#295) * Added CHANGELOG for v20190701 * Install ec2-instance-connect * refactor packer variables * Add c5.12xlarge and c5.24xlarge instances * Add new m5 and r5 instances * Fix t3a.small limit * add support for ap-east-1 region (awslabs#305) * 2107 allow private ssh when building (awslabs#303) * added a set of variables to allow private ssh to non-default vpc * make filepaths of ./files/ and install-worker relative to packer template dir * updated ami_description to a variable * change the amiName pattern to use minor version (awslabs#307) * update S3_URL_BASE environment variable in install-worker.sh * v20190814 release (awslabs#316) * Update list of instance types (awslabs#320) * Add all new instance types already added to the CNI * Add support for the u-*tb1.metal instances (Fix awslabs#319) * add support for me-south-1 region (awslabs#322) * Adding new directory and file for 1.14 and above by removing --allow-privileged=true flag (awslabs#327) * Add Change log for AMI Release v20190906 (awslabs#329) * sync nodegroup template to latest available (awslabs#335) * sync eks node group template to be latest available 1. add support to use ssm parameter for amiID 2. add support for all instance types supported by cni 3. formatted with rain(https://github.com/aws-cloudformation/rain) * add new CFN version 2019-09-17 * Add support for g4 instance family * Add G4DN instance family to node group template * Add change log for AMI Release v20190927 (awslabs#345) * Add 1.14 to the EKS Makefile and update older versions (awslabs#336) Add 1.14 to the list of Makefile targets. Remove 1.10 as it's no longer a supported version Update versions and build dates for older EKS versions * Add support for m5n/m5dn/r5n/r5dn instances * Remove snowflake for kubelet secret-polling config (awslabs#352) * Set a minimum evictionHard and kubeReserved * Output the autoscaling group name This name of the AutoScaling Group is useful for things like the Cluster Autoscaler so that it can manage automatic cluster scaling. * awslabs#361 - custom pause container image support (awslabs#362) * awslabs#361 - custom pause container image support * Set kubeReserved dynamically and evictionHard statically (awslabs#367) * Updating Docker version (awslabs#373) * Remove the ec2-net-utils package (awslabs#368) * Remove the ec2-net-utils package * Add code comment to describe the ec2-net-utils change * Make 'kube-bench' happy. Signed-off-by: Bruno Miguel Custódio <[email protected]> * add support for c5d.12x/c5d.24x/c5d.metal * Adding new instance types (m6g) (awslabs#378) * Revert "Make 'kube-bench' happy." since there are changes being concerned (awslabs#381) This reverts commit 593691e. * Fixed setting of DNS_CLUSTER_IP in bootstrap.sh (awslabs#226) * Replaced API calls for deciding DNS_CLUSTER_IP with arg * Bypass the metadata calls to avoid 404 errors * Fall back to MAC logic if --dns-cluster-ip is absent * Updated comment for --dns-cluster-ip * Support docker-in-docker by only returning the oldest dockerd process * TLS Ciphersuite: restrict to TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 See section 2.1.14 of the CIS benchmark: > [2.1.14] Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers > If using a Kubelet config file, edit the file to set TLSCipherSuites: to TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 > If using executable arguments, edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameter. > --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 Note that this is a regression, this had been set previously in PR awslabs#276 but got lost in awslabs#352. * Script for collecting window and ubuntu worker logs (awslabs#354) * Script for collecting window worker logs * Ubuntu support and directory re-org * Collect files from EKS logs folder * Updates to kubelet svc and kubeconfig * Updated Readme for Windows * add ability to specify aws_region & binary_bucket_region & source_ami_owners (awslabs#396) * adding support for china regions (awslabs#398) * kubelet.service should wait for iptables lock (awslabs#401) This commit makes kubelet.service wait up to 5 seconds for an iptables lock in the `ExecStartPre` step, instead of failing immediately if something else is holding the lock. * fix tls suit to be recommended by cis bench (awslabs#403) * Fix retries in bootstrap.sh If `aws eks describe-cluster` fails the first time, the retries never work because the `rc` value is never able to be set back to zero * update binaries to use latest ones (awslabs#408) * validate_yum (awslabs#411) * add ability to use precreated security group (awslabs#412) * add scripts folder (awslabs#413) * Remove invalid target 1.11 (awslabs#421) Currently, AWS EKS is no longer support Kubernetes 1.11 * Update install-worker.sh and eks-worker-al2.json (awslabs#402) * Update install-worker.sh and eks-worker-al2.json * Update kubelet.service * added ability to share amis in builder * added ability to share amis in builder * Rebasing from master * Added remote_folder to cleanup_additional_repos.sh provisioner * Added remote_folder to install_additional_repos.sh provisioner * Added remote_folder to validate.sh provisioner * adding support for 1.14 and updating cni * cni to v0.6.0 back - as v.0.7.1 has no binary * reverting back to plugins version * Remove mutating calls and ignore collection of unknown logs * Added 1.15 support and removed --allow-privileged flag from all EKS supported versions (1.12+). (awslabs#428) * Fix URL for 1.15 binaries (awslabs#429) * Fixed amazon-eks-nodegroup.yaml lint issues * Consistent Docker GID version in Image (awslabs#430) * Docker install across versions change GID for docker, this causes problems for consistency. This commit solves it by adding same GID to docker install * Docker install across versions change GID for docker, this causes problems for consistency. This commit solves it by adding same GID to docker install Co-authored-by: Janis Orlovs <[email protected]> * Move compressed file to /var/log (awslabs#436) * Force create the group id (awslabs#437) "the -f is force, -o is overwrite, meaning if there is an existing group with number 1950, it will create a new one with the name docker" * Fix useradd to run with privileges * Removing dependency on Authenticator binary (awslabs#440) * Reducing memory allocated in kubeReserved (awslabs#419) * Revert "Removing dependency on Authenticator binary (awslabs#440)" (awslabs#446) This reverts commit 4e0e916. * Adding support to upgrade kernel while building AMI (awslabs#447) * fix(amazon-eks-nodegroup): add ec2 service principals for isolated regions * Add inf1 instance family in EKS AMI packer configuration * Removed AssociatePublicIpAddress setting from NodeLaunchCongig and added NodeSecurityGroup dependency to SG Ingress/Egress (awslabs#450) Co-authored-by: Vishal Gupta <[email protected]> * added 1.15 * updated Jenkinsfile * updated kubelet latest from main source * typo * make kubelet service matches original master branch * Makefile updated * updated a few more * newline - yea newline * revert back to 1.15.10 * updated install-worker.sh Co-authored-by: Nithish <[email protected]> Co-authored-by: Hugo Ribeiro <[email protected]> Co-authored-by: M00nF1sh <[email protected]> Co-authored-by: Micah Hausler <[email protected]> Co-authored-by: Matthew Wong <[email protected]> Co-authored-by: Claes Mogren <[email protected]> Co-authored-by: wong yan yee <[email protected]> Co-authored-by: blakeroberts-wk <[email protected]> Co-authored-by: josselin-c <[email protected]> Co-authored-by: Bhagwat kumar Singh <[email protected]> Co-authored-by: Jiaxin Shan <[email protected]> Co-authored-by: Will Thames <[email protected]> Co-authored-by: Shyam JVS <[email protected]> Co-authored-by: Dwayne Bailey <[email protected]> Co-authored-by: Andrew Johnstone <[email protected]> Co-authored-by: natherz97 <[email protected]> Co-authored-by: Kausheel Kumar <[email protected]> Co-authored-by: Bruno Miguel Custódio <[email protected]> Co-authored-by: ajayk <[email protected]> Co-authored-by: sramabad1 <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Co-authored-by: Andrew Hemming <[email protected]> Co-authored-by: Eric Webster <[email protected]> Co-authored-by: Florent Delannoy <[email protected]> Co-authored-by: Arun Bhagyanath <[email protected]> Co-authored-by: Justin Owen <[email protected]> Co-authored-by: Aaron Ackerman <[email protected]> Co-authored-by: Tam Mach <[email protected]> Co-authored-by: zadowsmash <[email protected]> Co-authored-by: Shabir Ahmed <[email protected]> Co-authored-by: Abeer Sethi <[email protected]> Co-authored-by: Will Thames <[email protected]> Co-authored-by: Octavio Martin <[email protected]> Co-authored-by: Jānis Orlovs <[email protected]> Co-authored-by: Janis Orlovs <[email protected]> Co-authored-by: Divyesh Khandeshi <[email protected]> Co-authored-by: cmdallas <[email protected]> Co-authored-by: gaogilb <[email protected]> Co-authored-by: Vishal Gupta <[email protected]> Co-authored-by: Vishal Gupta <[email protected]>
sakomws
pushed a commit
to OpenGov/amazon-eks-ami
that referenced
this pull request
May 13, 2020
* Moving log collector script to Amazon eks ami repo (awslabs#243) * Moving log collector script to this repo * Added changes according to 1.4.1 * Update eks-log-collector.sh URL on readme The instructions on readme were still pointing at the original repository. Updating to reflect the new location * remove kubectl dependency (awslabs#295) * Added CHANGELOG for v20190701 * Install ec2-instance-connect * refactor packer variables * Add c5.12xlarge and c5.24xlarge instances * Add new m5 and r5 instances * Fix t3a.small limit * add support for ap-east-1 region (awslabs#305) * 2107 allow private ssh when building (awslabs#303) * added a set of variables to allow private ssh to non-default vpc * make filepaths of ./files/ and install-worker relative to packer template dir * updated ami_description to a variable * change the amiName pattern to use minor version (awslabs#307) * update S3_URL_BASE environment variable in install-worker.sh * v20190814 release (awslabs#316) * Update list of instance types (awslabs#320) * Add all new instance types already added to the CNI * Add support for the u-*tb1.metal instances (Fix awslabs#319) * add support for me-south-1 region (awslabs#322) * Adding new directory and file for 1.14 and above by removing --allow-privileged=true flag (awslabs#327) * Add Change log for AMI Release v20190906 (awslabs#329) * sync nodegroup template to latest available (awslabs#335) * sync eks node group template to be latest available 1. add support to use ssm parameter for amiID 2. add support for all instance types supported by cni 3. formatted with rain(https://github.com/aws-cloudformation/rain) * add new CFN version 2019-09-17 * Add support for g4 instance family * Add G4DN instance family to node group template * Add change log for AMI Release v20190927 (awslabs#345) * Add 1.14 to the EKS Makefile and update older versions (awslabs#336) Add 1.14 to the list of Makefile targets. Remove 1.10 as it's no longer a supported version Update versions and build dates for older EKS versions * Add support for m5n/m5dn/r5n/r5dn instances * Remove snowflake for kubelet secret-polling config (awslabs#352) * Set a minimum evictionHard and kubeReserved * Output the autoscaling group name This name of the AutoScaling Group is useful for things like the Cluster Autoscaler so that it can manage automatic cluster scaling. * awslabs#361 - custom pause container image support (awslabs#362) * awslabs#361 - custom pause container image support * Set kubeReserved dynamically and evictionHard statically (awslabs#367) * Updating Docker version (awslabs#373) * Remove the ec2-net-utils package (awslabs#368) * Remove the ec2-net-utils package * Add code comment to describe the ec2-net-utils change * Make 'kube-bench' happy. Signed-off-by: Bruno Miguel Custódio <[email protected]> * add support for c5d.12x/c5d.24x/c5d.metal * Adding new instance types (m6g) (awslabs#378) * Revert "Make 'kube-bench' happy." since there are changes being concerned (awslabs#381) This reverts commit 593691e. * Fixed setting of DNS_CLUSTER_IP in bootstrap.sh (awslabs#226) * Replaced API calls for deciding DNS_CLUSTER_IP with arg * Bypass the metadata calls to avoid 404 errors * Fall back to MAC logic if --dns-cluster-ip is absent * Updated comment for --dns-cluster-ip * Support docker-in-docker by only returning the oldest dockerd process * TLS Ciphersuite: restrict to TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 See section 2.1.14 of the CIS benchmark: > [2.1.14] Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers > If using a Kubelet config file, edit the file to set TLSCipherSuites: to TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 > If using executable arguments, edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameter. > --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 Note that this is a regression, this had been set previously in PR awslabs#276 but got lost in awslabs#352. * Script for collecting window and ubuntu worker logs (awslabs#354) * Script for collecting window worker logs * Ubuntu support and directory re-org * Collect files from EKS logs folder * Updates to kubelet svc and kubeconfig * Updated Readme for Windows * add ability to specify aws_region & binary_bucket_region & source_ami_owners (awslabs#396) * adding support for china regions (awslabs#398) * kubelet.service should wait for iptables lock (awslabs#401) This commit makes kubelet.service wait up to 5 seconds for an iptables lock in the `ExecStartPre` step, instead of failing immediately if something else is holding the lock. * fix tls suit to be recommended by cis bench (awslabs#403) * Fix retries in bootstrap.sh If `aws eks describe-cluster` fails the first time, the retries never work because the `rc` value is never able to be set back to zero * update binaries to use latest ones (awslabs#408) * validate_yum (awslabs#411) * add ability to use precreated security group (awslabs#412) * add scripts folder (awslabs#413) * Remove invalid target 1.11 (awslabs#421) Currently, AWS EKS is no longer support Kubernetes 1.11 * Update install-worker.sh and eks-worker-al2.json (awslabs#402) * Update install-worker.sh and eks-worker-al2.json * Update kubelet.service * added ability to share amis in builder * added ability to share amis in builder * Rebasing from master * Added remote_folder to cleanup_additional_repos.sh provisioner * Added remote_folder to install_additional_repos.sh provisioner * Added remote_folder to validate.sh provisioner * Remove mutating calls and ignore collection of unknown logs * Added 1.15 support and removed --allow-privileged flag from all EKS supported versions (1.12+). (awslabs#428) * Fix URL for 1.15 binaries (awslabs#429) * Fixed amazon-eks-nodegroup.yaml lint issues * Consistent Docker GID version in Image (awslabs#430) * Docker install across versions change GID for docker, this causes problems for consistency. This commit solves it by adding same GID to docker install * Docker install across versions change GID for docker, this causes problems for consistency. This commit solves it by adding same GID to docker install Co-authored-by: Janis Orlovs <[email protected]> * Move compressed file to /var/log (awslabs#436) * Force create the group id (awslabs#437) "the -f is force, -o is overwrite, meaning if there is an existing group with number 1950, it will create a new one with the name docker" * Fix useradd to run with privileges * Removing dependency on Authenticator binary (awslabs#440) * Reducing memory allocated in kubeReserved (awslabs#419) * Revert "Removing dependency on Authenticator binary (awslabs#440)" (awslabs#446) This reverts commit 4e0e916. * Adding support to upgrade kernel while building AMI (awslabs#447) * fix(amazon-eks-nodegroup): add ec2 service principals for isolated regions * Add inf1 instance family in EKS AMI packer configuration * Removed AssociatePublicIpAddress setting from NodeLaunchCongig and added NodeSecurityGroup dependency to SG Ingress/Egress (awslabs#450) Co-authored-by: Vishal Gupta <[email protected]> * Add a flag that allows CNI packages to be pulled from S3 instead of Github. (awslabs#457) The default behavior is unchanged and will still pull assets from Github. * update source AMI owner and ECR repo for govcloud (awslabs#458) * updated ipamd information files extension to json (awslabs#451) * updated ipamd data file extension to json * updated ipamd metrics file extension * Adding 1.16 to Makefile (awslabs#459) * downgrade * Add a new manifest containing the AMI name (awslabs#471) This commit adds a new manifest which contains AMI name in the manifest filename so that parallel builds can be triggered. Even though the new manifest is now generated along with the current one for backwards compatibility, eventually the old manifest (manifest.json) will be deprecated. * changelog updated * added udev setting * small updates * some fix * added udev again Co-authored-by: Nithish <[email protected]> Co-authored-by: Hugo Ribeiro <[email protected]> Co-authored-by: M00nF1sh <[email protected]> Co-authored-by: Micah Hausler <[email protected]> Co-authored-by: Matthew Wong <[email protected]> Co-authored-by: Claes Mogren <[email protected]> Co-authored-by: wong yan yee <[email protected]> Co-authored-by: blakeroberts-wk <[email protected]> Co-authored-by: josselin-c <[email protected]> Co-authored-by: Bhagwat kumar Singh <[email protected]> Co-authored-by: Jiaxin Shan <[email protected]> Co-authored-by: Will Thames <[email protected]> Co-authored-by: Shyam JVS <[email protected]> Co-authored-by: Dwayne Bailey <[email protected]> Co-authored-by: Andrew Johnstone <[email protected]> Co-authored-by: natherz97 <[email protected]> Co-authored-by: Kausheel Kumar <[email protected]> Co-authored-by: Bruno Miguel Custódio <[email protected]> Co-authored-by: ajayk <[email protected]> Co-authored-by: sramabad1 <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Co-authored-by: Andrew Hemming <[email protected]> Co-authored-by: Eric Webster <[email protected]> Co-authored-by: Florent Delannoy <[email protected]> Co-authored-by: Arun Bhagyanath <[email protected]> Co-authored-by: Justin Owen <[email protected]> Co-authored-by: Aaron Ackerman <[email protected]> Co-authored-by: Tam Mach <[email protected]> Co-authored-by: zadowsmash <[email protected]> Co-authored-by: Abeer Sethi <[email protected]> Co-authored-by: Will Thames <[email protected]> Co-authored-by: Octavio Martin <[email protected]> Co-authored-by: Jānis Orlovs <[email protected]> Co-authored-by: Janis Orlovs <[email protected]> Co-authored-by: Divyesh Khandeshi <[email protected]> Co-authored-by: cmdallas <[email protected]> Co-authored-by: gaogilb <[email protected]> Co-authored-by: Vishal Gupta <[email protected]> Co-authored-by: Vishal Gupta <[email protected]> Co-authored-by: Bronson Mirafuentes <[email protected]> Co-authored-by: Sai Teja Penugonda <[email protected]> Co-authored-by: Shabir Ahmed <[email protected]> Co-authored-by: Saurav Agarwalla <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes:
Moving the log collector script to this Amazon EKS AMI repository.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.