Fix KeyError: SecurityGroups in elasticache module. #410

stefanhorning · 2021-02-12T14:25:49Z

SUMMARY

Also improve docs a little.

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

elasticache module

ADDITIONAL INFORMATION

Module would otherwise throw Fix KeyError: 'SecurityGroups' when updating an existing Cache Cluster with a security group.

Also it was confusing to me at first if the security group param should be provided when in a VPC, as the cache_security_groups flag can't be used then and I didn't read further down the docs (on the Docs website). Hence I added a hint for others. But ideally this should be handled by the module internally, as it can be detected if VPC or not, based on the cache_subnet_group param (so only one SG param needs to be exposed to user).

…a little

ansibullbot · 2021-02-12T14:26:37Z

cc @alachaum @jillr @jsdalton @s-hertel @tremble @wimnat
click here for bot help

markuman

It would be nice if you can expand the integration test that it tests the SecurityGroup option of your PR.
Furthermore, you need a changelog fragment.

tests/integration/targets/elasticache/tasks/main.yml

stefanhorning · 2021-02-15T08:41:29Z

Sure, I was in the progress of looking into expanding the tests anyway.

… tests for Elasticache module.

tests/integration/targets/elasticache/tasks/main.yml

…n wasn't picke up from test wide config

tests/integration/targets/elasticache/tasks/main.yml

stefanhorning · 2021-02-15T10:23:19Z

Also I am getting this build error:

"/usr/local/lib/python2.7/dist-packages/OpenSSL/crypto.py:14: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.\n  from cryptography import utils, x509\nTraceback (most recent call last):\n  File \"/root/.ansible/tmp/ansible-tmp-1613384043.93-354-274813038104715/AnsiballZ_elasticache.py

Shouldn't we stop testing with Python 2.7, as it's super outdated an mist libraries start dropping support for it?

tremble · 2021-02-15T12:16:50Z

Shouldn't we stop testing with Python 2.7, as it's super outdated an mist libraries start dropping support for it.

We've mostly picked up our support 'requirements' from Ansible itself. Which will likely have some form of support for RHEL6 for another couple of years.

Because boto3 is dropping support for Python 2.7 I suspect we'll be releasing a 'major' version somewhere after July in which we drop official support for Python 2.7. ( https://boto3.amazonaws.com/v1/documentation/api/1.16.56/guide/migrationpy3.html )

…ms/credentials passing

…cache subnet group in another place

stefanhorning · 2021-02-16T16:01:52Z

Thanks for the tip @tremble, that seemed to have done the trick.

All code should be fixed now. Now it's just about fixing the CI setup to allow access for test user:

User: arn:aws:sts::966509639900:assumed-role/ansible-core-ci-test-prod/prod=shippable=ansible-collections=community.aws=1534.24 is not authorized to perform: elasticache:DescribeCacheClusters on resource: arn:aws:elasticache:us-east-1:966509639900:cluster:shippable-1534-24-elasticache-module-redis-test

markuman · 2021-02-16T16:14:34Z

@stefanhorning you need another PR in mattclay/aws-terminator
it's a service that deletes frequently all dangle services and resources.
I guess elasticache belongs to the data-services.yaml.
https://github.com/mattclay/aws-terminator/blob/master/aws/policy/data-services.yaml

Just reference in that PR to this PR.

… naming everywhere)

stefanhorning · 2021-02-17T09:57:35Z

I guess I addressed all issues now. So hopefully CI will turn green once mattclay/aws-terminator#126 is merged.

jillr

New policies and terminator will be deployed shortly, couple of things here I found while testing those.

tests/integration/targets/elasticache/tasks/main.yml

tests/integration/targets/elasticache/defaults/main.yml

tests/integration/targets/elasticache/tasks/main.yml

stefanhorning · 2021-02-26T11:41:46Z

@jillr I think I addressed all your comments and a few more issues. I hope tests will run through now. Once green this PR can be merged.

stefanhorning · 2021-02-26T12:00:20Z

Tests are running through now. Just get an error in one of the python versions (running in parallel to the others):

"code": "InsufficientCacheClusterCapacity", 
"message": "Cannot create a cache cluster because there is no availability zone corresponding to the subnets in the cache subnet group with sufficient capacity."

This is probably due to an AWS limit that get's hit if all the parallel tests try to create a cluster. @jillr Probably we need to increase the cache cluster limits for that AWS account.

tremble

Couple of minor tweaks.

The flake was due to Amazon capacity issues:
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/ErrorMessages.html#ErrorMessages.INSUFFICIENT_CACHE_CLUSTER_CAPACITY

changelogs/fragments/410-elasticache-fixes.yml

tests/integration/targets/elasticache/aliases

Co-authored-by: Mark Chappell <[email protected]>

jillr · 2021-04-19T22:52:10Z

recheck

jillr

Tests pass locally and LGTM once CI (excluding shippable) is green

tremble · 2021-04-20T09:01:27Z

recheck

…s/elasticache_fixes Fix KeyError: SecurityGroups in elasticache module. Reviewed-by: https://github.com/apps/ansible-zuul This commit was initially merged in https://github.com/ansible-collections/community.aws See: ansible-collections@dbf3697

…s/elasticache_fixes Fix KeyError: SecurityGroups in elasticache module. Reviewed-by: https://github.com/apps/ansible-zuul This commit was initially merged in https://github.com/ansible-collections/community.aws See: ansible-collections@fd0bcb5

…che_fixes Fix KeyError: SecurityGroups in elasticache module. Reviewed-by: https://github.com/apps/ansible-zuul

…_ec2_vpc_nat_gateway Migrate ec2_vpc_nat_gateway* modules and tests Migrate ec2_vpc_nat_gateway* modules and tests Reviewed-by: Jill R <None> Reviewed-by: Alina Buzachis <None> Reviewed-by: Gonéri Le Bouder <[email protected]> Reviewed-by: Mark Chappell <None> Reviewed-by: Mike Graves <[email protected]> Reviewed-by: None <None>

Fix KeyError: SecurityGroup in elasticache module. Also improve docs …

7251f4c

…a little

Shorter line

6ab3f02

ansibullbot added community_review and removed small_patch Hopefully easy to review needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR labels Feb 12, 2021

Added simple integration test for elasticache module

cd9c9fd

ansibullbot added integration tests/integration needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR tests tests and removed community_review labels Feb 12, 2021

markuman requested changes Feb 12, 2021

View reviewed changes

tests/integration/targets/elasticache/tasks/main.yml Show resolved Hide resolved

tests/integration/targets/elasticache/tasks/main.yml Show resolved Hide resolved

stefanhorning added 3 commits February 15, 2021 10:34

Add changelog fragment for Elasticache module change. Add integration…

50c8d70

… tests for Elasticache module.

Complete cleanup all integration test resources for elasticache

7d6ecb7

Fix variable names

566bd22

stefanhorning commented Feb 15, 2021

View reviewed changes

tests/integration/targets/elasticache/tasks/main.yml Outdated Show resolved Hide resolved

Attach region flag to all tasks in integration test, as somehow regio…

717f51c

…n wasn't picke up from test wide config

stefanhorning commented Feb 15, 2021

View reviewed changes

tests/integration/targets/elasticache/tasks/main.yml Show resolved Hide resolved

ansibullbot removed the needs_triage label Feb 15, 2021

stefanhorning added 2 commits February 16, 2021 16:10

Remove FQDN from modules names in tests in an attempt to fix AWS para…

2562343

…ms/credentials passing

Also prefix vpc in test with resource_prefix. Use variable for elasti…

1f85d11

…cache subnet group in another place

stefanhorning mentioned this pull request Feb 17, 2021

Allow a bunch of elasticache actions for CI testing elasticache module mattclay/aws-terminator#126

Merged

More improvements in elasticache integration tests (use variables for…

e59b5f9

… naming everywhere)

jillr requested changes Feb 26, 2021

View reviewed changes

stefanhorning added 3 commits February 26, 2021 11:47

Fixes in elasticache integration tests

9608caf

Fix elasticache integration test VPC clanup

8c385f4

Fix elasticache integration test syntax

bdabad5

tremble approved these changes Mar 16, 2021

View reviewed changes

changelogs/fragments/410-elasticache-fixes.yml Outdated Show resolved Hide resolved

changelogs/fragments/410-elasticache-fixes.yml Outdated Show resolved Hide resolved

tests/integration/targets/elasticache/aliases Outdated Show resolved Hide resolved

stefanhorning and others added 3 commits March 22, 2021 09:47

Update changelogs/fragments/410-elasticache-fixes.yml

d787649

Co-authored-by: Mark Chappell <[email protected]>

Update changelogs/fragments/410-elasticache-fixes.yml

a859180

Co-authored-by: Mark Chappell <[email protected]>

Update tests/integration/targets/elasticache/aliases

c233496

Co-authored-by: Mark Chappell <[email protected]>

jillr approved these changes Apr 19, 2021

View reviewed changes

jillr added the gate label Apr 20, 2021

ansible-zuul bot merged commit dbf3697 into ansible-collections:main Apr 20, 2021

alinabuzachis mentioned this pull request Jul 20, 2021

Migrate ec2_vpc_nat_gateway* modules and tests #652

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix KeyError: SecurityGroups in elasticache module. #410

Fix KeyError: SecurityGroups in elasticache module. #410

stefanhorning commented Feb 12, 2021 •

edited

Loading

ansibullbot commented Feb 12, 2021

markuman left a comment •

edited

Loading

stefanhorning commented Feb 15, 2021 •

edited

Loading

stefanhorning commented Feb 15, 2021 •

edited

Loading

tremble commented Feb 15, 2021

stefanhorning commented Feb 16, 2021 •

edited

Loading

markuman commented Feb 16, 2021 •

edited

Loading

stefanhorning commented Feb 17, 2021

jillr left a comment

stefanhorning commented Feb 26, 2021 •

edited

Loading

stefanhorning commented Feb 26, 2021

tremble left a comment

jillr commented Apr 19, 2021 •

edited

Loading

jillr left a comment

tremble commented Apr 20, 2021

Fix KeyError: SecurityGroups in elasticache module. #410

Fix KeyError: SecurityGroups in elasticache module. #410

Conversation

stefanhorning commented Feb 12, 2021 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

ansibullbot commented Feb 12, 2021

markuman left a comment • edited Loading

Choose a reason for hiding this comment

stefanhorning commented Feb 15, 2021 • edited Loading

stefanhorning commented Feb 15, 2021 • edited Loading

tremble commented Feb 15, 2021

stefanhorning commented Feb 16, 2021 • edited Loading

markuman commented Feb 16, 2021 • edited Loading

stefanhorning commented Feb 17, 2021

jillr left a comment

Choose a reason for hiding this comment

stefanhorning commented Feb 26, 2021 • edited Loading

stefanhorning commented Feb 26, 2021

tremble left a comment

Choose a reason for hiding this comment

jillr commented Apr 19, 2021 • edited Loading

jillr left a comment

Choose a reason for hiding this comment

tremble commented Apr 20, 2021

stefanhorning commented Feb 12, 2021 •

edited

Loading

markuman left a comment •

edited

Loading

stefanhorning commented Feb 15, 2021 •

edited

Loading

stefanhorning commented Feb 15, 2021 •

edited

Loading

stefanhorning commented Feb 16, 2021 •

edited

Loading

markuman commented Feb 16, 2021 •

edited

Loading

stefanhorning commented Feb 26, 2021 •

edited

Loading

jillr commented Apr 19, 2021 •

edited

Loading