Remove the ec2-net-utils package #368
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mogren
suggested changes
Nov 4, 2019
Comment on lines
+67
to
+68
| if yum list installed | grep ec2-net-utils; then sudo yum remove ec2-net-utils -y -q; fi | ||
|
|
There was a problem hiding this comment.
Thanks, it is true that this package causes problems with the AWS CNI. Could you please add a comment before this line to explain why we remove it.
|
I've added a 1 line code commit now. If you'd like me to be more descriptive, let me know and I'll be happy to do so. If you want, I can include the full paragraph explanation above if you think that level of detail would be important in the commit. Otherwise, for now, I've just added the 1 line explanation. |
|
@Kausheel thanks for adding this :) This package wasted a lot of time while debugging issue with our image few months ago. It never came to my mind that we can check if it's installed and then remove it like this. |
shabir61
added a commit
to OpenGov/amazon-eks-ami
that referenced
this pull request
Apr 20, 2020
* Moving log collector script to Amazon eks ami repo (awslabs#243) * Moving log collector script to this repo * Added changes according to 1.4.1 * Update eks-log-collector.sh URL on readme The instructions on readme were still pointing at the original repository. Updating to reflect the new location * remove kubectl dependency (awslabs#295) * Added CHANGELOG for v20190701 * Install ec2-instance-connect * refactor packer variables * Add c5.12xlarge and c5.24xlarge instances * Add new m5 and r5 instances * Fix t3a.small limit * add support for ap-east-1 region (awslabs#305) * 2107 allow private ssh when building (awslabs#303) * added a set of variables to allow private ssh to non-default vpc * make filepaths of ./files/ and install-worker relative to packer template dir * updated ami_description to a variable * change the amiName pattern to use minor version (awslabs#307) * update S3_URL_BASE environment variable in install-worker.sh * v20190814 release (awslabs#316) * Update list of instance types (awslabs#320) * Add all new instance types already added to the CNI * Add support for the u-*tb1.metal instances (Fix awslabs#319) * add support for me-south-1 region (awslabs#322) * Adding new directory and file for 1.14 and above by removing --allow-privileged=true flag (awslabs#327) * Add Change log for AMI Release v20190906 (awslabs#329) * sync nodegroup template to latest available (awslabs#335) * sync eks node group template to be latest available 1. add support to use ssm parameter for amiID 2. add support for all instance types supported by cni 3. formatted with rain(https://github.com/aws-cloudformation/rain) * add new CFN version 2019-09-17 * Add support for g4 instance family * Add G4DN instance family to node group template * Add change log for AMI Release v20190927 (awslabs#345) * Add 1.14 to the EKS Makefile and update older versions (awslabs#336) Add 1.14 to the list of Makefile targets. Remove 1.10 as it's no longer a supported version Update versions and build dates for older EKS versions * Add support for m5n/m5dn/r5n/r5dn instances * Remove snowflake for kubelet secret-polling config (awslabs#352) * Set a minimum evictionHard and kubeReserved * Output the autoscaling group name This name of the AutoScaling Group is useful for things like the Cluster Autoscaler so that it can manage automatic cluster scaling. * awslabs#361 - custom pause container image support (awslabs#362) * awslabs#361 - custom pause container image support * Set kubeReserved dynamically and evictionHard statically (awslabs#367) * Updating Docker version (awslabs#373) * Remove the ec2-net-utils package (awslabs#368) * Remove the ec2-net-utils package * Add code comment to describe the ec2-net-utils change * Make 'kube-bench' happy. Signed-off-by: Bruno Miguel Custódio <[email protected]> * add support for c5d.12x/c5d.24x/c5d.metal * Adding new instance types (m6g) (awslabs#378) * Revert "Make 'kube-bench' happy." since there are changes being concerned (awslabs#381) This reverts commit 593691e. * Fixed setting of DNS_CLUSTER_IP in bootstrap.sh (awslabs#226) * Replaced API calls for deciding DNS_CLUSTER_IP with arg * Bypass the metadata calls to avoid 404 errors * Fall back to MAC logic if --dns-cluster-ip is absent * Updated comment for --dns-cluster-ip * Support docker-in-docker by only returning the oldest dockerd process * TLS Ciphersuite: restrict to TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 See section 2.1.14 of the CIS benchmark: > [2.1.14] Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers > If using a Kubelet config file, edit the file to set TLSCipherSuites: to TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 > If using executable arguments, edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameter. > --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 Note that this is a regression, this had been set previously in PR awslabs#276 but got lost in awslabs#352. * Script for collecting window and ubuntu worker logs (awslabs#354) * Script for collecting window worker logs * Ubuntu support and directory re-org * Collect files from EKS logs folder * Updates to kubelet svc and kubeconfig * Updated Readme for Windows * add ability to specify aws_region & binary_bucket_region & source_ami_owners (awslabs#396) * adding support for china regions (awslabs#398) * kubelet.service should wait for iptables lock (awslabs#401) This commit makes kubelet.service wait up to 5 seconds for an iptables lock in the `ExecStartPre` step, instead of failing immediately if something else is holding the lock. * fix tls suit to be recommended by cis bench (awslabs#403) * Fix retries in bootstrap.sh If `aws eks describe-cluster` fails the first time, the retries never work because the `rc` value is never able to be set back to zero * update binaries to use latest ones (awslabs#408) * validate_yum (awslabs#411) * add ability to use precreated security group (awslabs#412) * add scripts folder (awslabs#413) * Remove invalid target 1.11 (awslabs#421) Currently, AWS EKS is no longer support Kubernetes 1.11 * Update install-worker.sh and eks-worker-al2.json (awslabs#402) * Update install-worker.sh and eks-worker-al2.json * Update kubelet.service * added ability to share amis in builder * added ability to share amis in builder * Rebasing from master * Added remote_folder to cleanup_additional_repos.sh provisioner * Added remote_folder to install_additional_repos.sh provisioner * Added remote_folder to validate.sh provisioner * adding support for 1.14 and updating cni * cni to v0.6.0 back - as v.0.7.1 has no binary * reverting back to plugins version * Remove mutating calls and ignore collection of unknown logs * Added 1.15 support and removed --allow-privileged flag from all EKS supported versions (1.12+). (awslabs#428) * Fix URL for 1.15 binaries (awslabs#429) * Fixed amazon-eks-nodegroup.yaml lint issues * Consistent Docker GID version in Image (awslabs#430) * Docker install across versions change GID for docker, this causes problems for consistency. This commit solves it by adding same GID to docker install * Docker install across versions change GID for docker, this causes problems for consistency. This commit solves it by adding same GID to docker install Co-authored-by: Janis Orlovs <[email protected]> * Move compressed file to /var/log (awslabs#436) * Force create the group id (awslabs#437) "the -f is force, -o is overwrite, meaning if there is an existing group with number 1950, it will create a new one with the name docker" * Fix useradd to run with privileges * Removing dependency on Authenticator binary (awslabs#440) * Reducing memory allocated in kubeReserved (awslabs#419) * Revert "Removing dependency on Authenticator binary (awslabs#440)" (awslabs#446) This reverts commit 4e0e916. * Adding support to upgrade kernel while building AMI (awslabs#447) * fix(amazon-eks-nodegroup): add ec2 service principals for isolated regions * Add inf1 instance family in EKS AMI packer configuration * Removed AssociatePublicIpAddress setting from NodeLaunchCongig and added NodeSecurityGroup dependency to SG Ingress/Egress (awslabs#450) Co-authored-by: Vishal Gupta <[email protected]> * added 1.15 * updated Jenkinsfile * updated kubelet latest from main source * typo * make kubelet service matches original master branch * Makefile updated * updated a few more * newline - yea newline * revert back to 1.15.10 * updated install-worker.sh Co-authored-by: Nithish <[email protected]> Co-authored-by: Hugo Ribeiro <[email protected]> Co-authored-by: M00nF1sh <[email protected]> Co-authored-by: Micah Hausler <[email protected]> Co-authored-by: Matthew Wong <[email protected]> Co-authored-by: Claes Mogren <[email protected]> Co-authored-by: wong yan yee <[email protected]> Co-authored-by: blakeroberts-wk <[email protected]> Co-authored-by: josselin-c <[email protected]> Co-authored-by: Bhagwat kumar Singh <[email protected]> Co-authored-by: Jiaxin Shan <[email protected]> Co-authored-by: Will Thames <[email protected]> Co-authored-by: Shyam JVS <[email protected]> Co-authored-by: Dwayne Bailey <[email protected]> Co-authored-by: Andrew Johnstone <[email protected]> Co-authored-by: natherz97 <[email protected]> Co-authored-by: Kausheel Kumar <[email protected]> Co-authored-by: Bruno Miguel Custódio <[email protected]> Co-authored-by: ajayk <[email protected]> Co-authored-by: sramabad1 <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Co-authored-by: Andrew Hemming <[email protected]> Co-authored-by: Eric Webster <[email protected]> Co-authored-by: Florent Delannoy <[email protected]> Co-authored-by: Arun Bhagyanath <[email protected]> Co-authored-by: Justin Owen <[email protected]> Co-authored-by: Aaron Ackerman <[email protected]> Co-authored-by: Tam Mach <[email protected]> Co-authored-by: zadowsmash <[email protected]> Co-authored-by: Shabir Ahmed <[email protected]> Co-authored-by: Abeer Sethi <[email protected]> Co-authored-by: Will Thames <[email protected]> Co-authored-by: Octavio Martin <[email protected]> Co-authored-by: Jānis Orlovs <[email protected]> Co-authored-by: Janis Orlovs <[email protected]> Co-authored-by: Divyesh Khandeshi <[email protected]> Co-authored-by: cmdallas <[email protected]> Co-authored-by: gaogilb <[email protected]> Co-authored-by: Vishal Gupta <[email protected]> Co-authored-by: Vishal Gupta <[email protected]>
sakomws
pushed a commit
to OpenGov/amazon-eks-ami
that referenced
this pull request
May 13, 2020
* Moving log collector script to Amazon eks ami repo (awslabs#243) * Moving log collector script to this repo * Added changes according to 1.4.1 * Update eks-log-collector.sh URL on readme The instructions on readme were still pointing at the original repository. Updating to reflect the new location * remove kubectl dependency (awslabs#295) * Added CHANGELOG for v20190701 * Install ec2-instance-connect * refactor packer variables * Add c5.12xlarge and c5.24xlarge instances * Add new m5 and r5 instances * Fix t3a.small limit * add support for ap-east-1 region (awslabs#305) * 2107 allow private ssh when building (awslabs#303) * added a set of variables to allow private ssh to non-default vpc * make filepaths of ./files/ and install-worker relative to packer template dir * updated ami_description to a variable * change the amiName pattern to use minor version (awslabs#307) * update S3_URL_BASE environment variable in install-worker.sh * v20190814 release (awslabs#316) * Update list of instance types (awslabs#320) * Add all new instance types already added to the CNI * Add support for the u-*tb1.metal instances (Fix awslabs#319) * add support for me-south-1 region (awslabs#322) * Adding new directory and file for 1.14 and above by removing --allow-privileged=true flag (awslabs#327) * Add Change log for AMI Release v20190906 (awslabs#329) * sync nodegroup template to latest available (awslabs#335) * sync eks node group template to be latest available 1. add support to use ssm parameter for amiID 2. add support for all instance types supported by cni 3. formatted with rain(https://github.com/aws-cloudformation/rain) * add new CFN version 2019-09-17 * Add support for g4 instance family * Add G4DN instance family to node group template * Add change log for AMI Release v20190927 (awslabs#345) * Add 1.14 to the EKS Makefile and update older versions (awslabs#336) Add 1.14 to the list of Makefile targets. Remove 1.10 as it's no longer a supported version Update versions and build dates for older EKS versions * Add support for m5n/m5dn/r5n/r5dn instances * Remove snowflake for kubelet secret-polling config (awslabs#352) * Set a minimum evictionHard and kubeReserved * Output the autoscaling group name This name of the AutoScaling Group is useful for things like the Cluster Autoscaler so that it can manage automatic cluster scaling. * awslabs#361 - custom pause container image support (awslabs#362) * awslabs#361 - custom pause container image support * Set kubeReserved dynamically and evictionHard statically (awslabs#367) * Updating Docker version (awslabs#373) * Remove the ec2-net-utils package (awslabs#368) * Remove the ec2-net-utils package * Add code comment to describe the ec2-net-utils change * Make 'kube-bench' happy. Signed-off-by: Bruno Miguel Custódio <[email protected]> * add support for c5d.12x/c5d.24x/c5d.metal * Adding new instance types (m6g) (awslabs#378) * Revert "Make 'kube-bench' happy." since there are changes being concerned (awslabs#381) This reverts commit 593691e. * Fixed setting of DNS_CLUSTER_IP in bootstrap.sh (awslabs#226) * Replaced API calls for deciding DNS_CLUSTER_IP with arg * Bypass the metadata calls to avoid 404 errors * Fall back to MAC logic if --dns-cluster-ip is absent * Updated comment for --dns-cluster-ip * Support docker-in-docker by only returning the oldest dockerd process * TLS Ciphersuite: restrict to TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 See section 2.1.14 of the CIS benchmark: > [2.1.14] Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers > If using a Kubelet config file, edit the file to set TLSCipherSuites: to TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 > If using executable arguments, edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameter. > --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 Note that this is a regression, this had been set previously in PR awslabs#276 but got lost in awslabs#352. * Script for collecting window and ubuntu worker logs (awslabs#354) * Script for collecting window worker logs * Ubuntu support and directory re-org * Collect files from EKS logs folder * Updates to kubelet svc and kubeconfig * Updated Readme for Windows * add ability to specify aws_region & binary_bucket_region & source_ami_owners (awslabs#396) * adding support for china regions (awslabs#398) * kubelet.service should wait for iptables lock (awslabs#401) This commit makes kubelet.service wait up to 5 seconds for an iptables lock in the `ExecStartPre` step, instead of failing immediately if something else is holding the lock. * fix tls suit to be recommended by cis bench (awslabs#403) * Fix retries in bootstrap.sh If `aws eks describe-cluster` fails the first time, the retries never work because the `rc` value is never able to be set back to zero * update binaries to use latest ones (awslabs#408) * validate_yum (awslabs#411) * add ability to use precreated security group (awslabs#412) * add scripts folder (awslabs#413) * Remove invalid target 1.11 (awslabs#421) Currently, AWS EKS is no longer support Kubernetes 1.11 * Update install-worker.sh and eks-worker-al2.json (awslabs#402) * Update install-worker.sh and eks-worker-al2.json * Update kubelet.service * added ability to share amis in builder * added ability to share amis in builder * Rebasing from master * Added remote_folder to cleanup_additional_repos.sh provisioner * Added remote_folder to install_additional_repos.sh provisioner * Added remote_folder to validate.sh provisioner * Remove mutating calls and ignore collection of unknown logs * Added 1.15 support and removed --allow-privileged flag from all EKS supported versions (1.12+). (awslabs#428) * Fix URL for 1.15 binaries (awslabs#429) * Fixed amazon-eks-nodegroup.yaml lint issues * Consistent Docker GID version in Image (awslabs#430) * Docker install across versions change GID for docker, this causes problems for consistency. This commit solves it by adding same GID to docker install * Docker install across versions change GID for docker, this causes problems for consistency. This commit solves it by adding same GID to docker install Co-authored-by: Janis Orlovs <[email protected]> * Move compressed file to /var/log (awslabs#436) * Force create the group id (awslabs#437) "the -f is force, -o is overwrite, meaning if there is an existing group with number 1950, it will create a new one with the name docker" * Fix useradd to run with privileges * Removing dependency on Authenticator binary (awslabs#440) * Reducing memory allocated in kubeReserved (awslabs#419) * Revert "Removing dependency on Authenticator binary (awslabs#440)" (awslabs#446) This reverts commit 4e0e916. * Adding support to upgrade kernel while building AMI (awslabs#447) * fix(amazon-eks-nodegroup): add ec2 service principals for isolated regions * Add inf1 instance family in EKS AMI packer configuration * Removed AssociatePublicIpAddress setting from NodeLaunchCongig and added NodeSecurityGroup dependency to SG Ingress/Egress (awslabs#450) Co-authored-by: Vishal Gupta <[email protected]> * Add a flag that allows CNI packages to be pulled from S3 instead of Github. (awslabs#457) The default behavior is unchanged and will still pull assets from Github. * update source AMI owner and ECR repo for govcloud (awslabs#458) * updated ipamd information files extension to json (awslabs#451) * updated ipamd data file extension to json * updated ipamd metrics file extension * Adding 1.16 to Makefile (awslabs#459) * downgrade * Add a new manifest containing the AMI name (awslabs#471) This commit adds a new manifest which contains AMI name in the manifest filename so that parallel builds can be triggered. Even though the new manifest is now generated along with the current one for backwards compatibility, eventually the old manifest (manifest.json) will be deprecated. * changelog updated * added udev setting * small updates * some fix * added udev again Co-authored-by: Nithish <[email protected]> Co-authored-by: Hugo Ribeiro <[email protected]> Co-authored-by: M00nF1sh <[email protected]> Co-authored-by: Micah Hausler <[email protected]> Co-authored-by: Matthew Wong <[email protected]> Co-authored-by: Claes Mogren <[email protected]> Co-authored-by: wong yan yee <[email protected]> Co-authored-by: blakeroberts-wk <[email protected]> Co-authored-by: josselin-c <[email protected]> Co-authored-by: Bhagwat kumar Singh <[email protected]> Co-authored-by: Jiaxin Shan <[email protected]> Co-authored-by: Will Thames <[email protected]> Co-authored-by: Shyam JVS <[email protected]> Co-authored-by: Dwayne Bailey <[email protected]> Co-authored-by: Andrew Johnstone <[email protected]> Co-authored-by: natherz97 <[email protected]> Co-authored-by: Kausheel Kumar <[email protected]> Co-authored-by: Bruno Miguel Custódio <[email protected]> Co-authored-by: ajayk <[email protected]> Co-authored-by: sramabad1 <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Co-authored-by: Andrew Hemming <[email protected]> Co-authored-by: Eric Webster <[email protected]> Co-authored-by: Florent Delannoy <[email protected]> Co-authored-by: Arun Bhagyanath <[email protected]> Co-authored-by: Justin Owen <[email protected]> Co-authored-by: Aaron Ackerman <[email protected]> Co-authored-by: Tam Mach <[email protected]> Co-authored-by: zadowsmash <[email protected]> Co-authored-by: Abeer Sethi <[email protected]> Co-authored-by: Will Thames <[email protected]> Co-authored-by: Octavio Martin <[email protected]> Co-authored-by: Jānis Orlovs <[email protected]> Co-authored-by: Janis Orlovs <[email protected]> Co-authored-by: Divyesh Khandeshi <[email protected]> Co-authored-by: cmdallas <[email protected]> Co-authored-by: gaogilb <[email protected]> Co-authored-by: Vishal Gupta <[email protected]> Co-authored-by: Vishal Gupta <[email protected]> Co-authored-by: Bronson Mirafuentes <[email protected]> Co-authored-by: Sai Teja Penugonda <[email protected]> Co-authored-by: Shabir Ahmed <[email protected]> Co-authored-by: Saurav Agarwalla <[email protected]>
@Kausheel We are currently debating over an effort to bring back the ec2-net-utils into the ami. Could you recollect the issues that you observed back then? We would like to make sure we are covering those issues in the tests. |
|
I've reached out to you offline with some more information. The summary of the original problem is that ec2-net-utils would try to apply entries in the Linux routing table which override/interfere with the routes created by the CNI. Pod networking would then break. You can test this by launching a Node without the package, deploy a Pod and verify network to/from the Pod is working. Check the state of the routing table entries. Then install ec2-net-utils, and wait for it to execute (/var/log/messages will have an entry when ec2-net-utils is triggered, which might happen on a DHCP lease refresh). Once the program executes, check the state of Pod networking again, and check for changes in the routing tables. We should observe that the routes managing traffic to/from the Pod have been modified by ec2-net-utils. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes:
I've encountered an issue when using Packer with the following AMI filter: amzn2-ami-hvm-*
The above filter allows users to use the "full" Amazon Linux 2 AMI (not the minimal AMI, or EKS optimized AMI).
The problem is that the full AM2 AMI contains the ec2-net-utils package. This package contains scripts to automatically create route table entries, which interferes with Pod networking. In my tests, the ec2-net-utils package would create route entries in the "Local" Route Table to direct Pod traffic to an incorrect interface. We want the CNI to take care of setting up the route entries, and this package conflicts with that.
The default AMI filter "amzn2-ami-minimal-hvm-*" and also the EKS optimized AMIs don't contain the ec2-net-utils package, so it's clearly not required for the Worker Nodes to function correctly.
This Pull Request will check to see if the ec2-net-utils package is installed, and if yes, it removes it. I've tested that Packer completes the build successfully with the change included. I've launched some Worker Nodes to verify that the package is successfully removed by checking /var/log/yum.log, and also by running
yum list installed.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.