Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s3_bucket transfer accleration #1

Open
wants to merge 42 commits into
base: main
Choose a base branch
from
Open

s3_bucket transfer accleration #1

wants to merge 42 commits into from

Conversation

abraverm
Copy link
Owner

@abraverm abraverm commented Apr 9, 2024

@abraverm abraverm force-pushed the accelerated branch 2 times, most recently from 381af42 to 7e656d5 Compare May 2, 2024 23:38
alinabuzachis and others added 28 commits May 6, 2024 13:08
… >= 1.29.0. (ansible-collections#2073)

Remove any requirement for botocore < 1.29.0

SUMMARY

Remove any requirement for botocore < 1.29.0

ISSUE TYPE


Docs Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Bikouo Aubin
Reviewed-by: Mark Chappell
…e-collections#2072)

Update return block in documentation for ec2 modules (part 1)

SUMMARY


Refer ansible-collections#1983
This PR aligns the RETURN blocks in the module documentation to accurately reflect what the module returns, ensuring consistency.
Modules updated:
ec2_eip_info, ec2_ami, ec2_ami_info, ec2_eni, ec2_eni_info, ec2_instance, ec2_instance_info, ec2_security_group, ec2_security_group_info
ISSUE TYPE


Docs Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Alina Buzachis
Reviewed-by: Mark Chappell
Reviewed-by: Mike Graves <[email protected]>
…lections#2078)

[main] 7.6.0 changelog

SUMMARY
Merge in the 7.6.0 changelog
ISSUE TYPE

Docs Pull Request

COMPONENT NAME
CHANGELOG.rst
changelogs/changelog.yaml
ADDITIONAL INFORMATION
See also ansible-collections#2075

Reviewed-by: Alina Buzachis
…ort (ansible-collections#2089)

RDS Integration tests - Bump MySQL/MariaDB versions to match AWS support

SUMMARY
Bump MySQL/MariaDB versions to match AWS support.  AWS dropped the versions we were using in our integration tests, bump them
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
rds_instance
rds_option_group
ADDITIONAL INFORMATION

Reviewed-by: Bikouo Aubin
…sible-collections#2025)

lambda_event - add support for maximum_batching_window_in_seconds

SUMMARY

Closes ansible-collections#1995
Fix when batch_size is greater than 10, by enabling support for setting maximum_batching_window_in_seconds.

ISSUE TYPE


Feature Pull Request

COMPONENT NAME

lambda_event

Reviewed-by: Mandar Kulkarni <[email protected]>
Reviewed-by: Bikouo Aubin
Reviewed-by: Alina Buzachis
Reviewed-by: Mark Chappell
Reviewed-by: Helen Bailey <[email protected]>
Bump release for main branch to 9.0.0-dev0

SUMMARY
Bump release for main branch to 9.0.0-dev0
ISSUE TYPE

Feature Pull Request

COMPONENT NAME
galaxy.yml
plugins/module_utils/common.py
tests/sanity/
ADDITIONAL INFORMATION

Reviewed-by: Mark Chappell
Drop support for ansible-core < 2.15

SUMMARY

Drop support for ansible-core < 2.15. It modifies the already existing changelog.

ISSUE TYPE


Bugfix Pull Request
Docs Pull Request
Feature Pull Request
New Module Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: GomathiselviS
Reviewed-by: Mark Chappell
CONTRIBUTING.md: fix hyperlink

SUMMARY

updated link for Guidelines for Ansible Amazon AWS module development
from
https://docs.ansible.com/ansible/latest/dev_guide/platforms/aws_guidelines.html
to
https://docs.ansible.com/ansible/latest/collections/amazon/aws/docsite/dev_guidelines.html

ISSUE TYPE


Docs Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Mike Graves <[email protected]>
…#2100)

Update main branch CHANGELOG after 8.0.0 release

SUMMARY

Update main branch CHANGELOG after 8.0.0 release

ISSUE TYPE


Docs Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Mike Graves <[email protected]>
Reviewed-by: Helen Bailey <[email protected]>
Update return block of ec2 modules (part 2)

SUMMARY


Refer ansible-collections#1983
This PR aligns the RETURN blocks in the module documentation to accurately reflect what the module returns, ensuring consistency.
Modules updated:
ec2_snapshot ec2_snapshot_info ec2_spot_instance ec2_vol ec2_vpc_endpoint ec2_vpc_endpoint_info ec2_vpc_endpoint_service_info ec2_vpc_nat_gateway ec2_vpc_nat_gateway_info ec2_vpc_route_table ec2_vpc_subnet ec2_vpc_subnet_info
ISSUE TYPE


Docs Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Alina Buzachis
Reviewed-by: GomathiselviS
Reviewed-by: Bikouo Aubin
Reviewed-by: Mike Graves <[email protected]>
…lections#2108)

s3_object - fix MemoryError when downloading large files

SUMMARY
fixes: ansible-collections#2107
The refactor in ansible-collections#1139 is triggering a full download of the file into memory when downloading files, this downloaded content was then being thrown away.
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
s3_object
ADDITIONAL INFORMATION

Reviewed-by: Alina Buzachis
Reviewed-by: Bikouo Aubin
…ats (ansible-collections#2080)

[Part #1]: Ensures the modules documentation follows the correct formats

SUMMARY

Ensures the modules documentation follows the correct formats https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_documenting.html#semantic-markup-within-module-documentation
More PRs are yet to come

ISSUE TYPE


Docs Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Mark Chappell
Reviewed-by: Bikouo Aubin
…le-collections#2105)

backup_vault: remove unnecessary return values from code block

SUMMARY

Removed unnecessary return values from code block
These parameters are not present in response syntax of create_backup_vault and describe_backup_vault
and are never returned by API.
optional_vals = set(
            [
                "S3KeyPrefix",
                "SnsTopicName",
                "SnsTopicARN",
                "CloudWatchLogsLogGroupArn",
                "CloudWatchLogsRoleArn",
                "KmsKeyId",
            ]
        )


ISSUE TYPE

COMPONENT NAME

backup_vault
ADDITIONAL INFORMATION

Reviewed-by: Alina Buzachis
Reviewed-by: GomathiselviS
…ats (ansible-collections#2081)

[Part #2]: Ensures the modules documentation follows the correct formats

SUMMARY


ISSUE TYPE


Docs Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Bikouo Aubin
Reviewed-by: Mark Chappell
…ts (ansible-collections#2082)

[Part #3] Ensures the modules documentation follows the correct formats

SUMMARY


ISSUE TYPE


Bugfix Pull Request
Docs Pull Request
Feature Pull Request
New Module Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Mark Chappell
…ied (ansible-collections#1979)

Don't ignore IPv6 addresses when a single network interface is specified

SUMMARY

When "ipv6_adresses" are used with "network" without specifying "interfaces", the parameter is ignored and no IPv6 adresses get set. I don't believe this is the intended or correct behaviour.

ISSUE TYPE


Bugfix Pull Request

COMPONENT NAME

ec2_instance
ADDITIONAL INFORMATION

Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Bikouo Aubin
Reviewed-by: Alina Buzachis
…ansible-collections#2111)

cloudwatch_metric_alarm: include evaluate_low_sample_count_percentile

SUMMARY


Fixes ansible-collections#1940
This PR adds evaluate_low_sample_count_percentile  parameter
ISSUE TYPE



Feature Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Mandar Kulkarni <[email protected]>
Reviewed-by: Bikouo Aubin
Update return blocks of cloud* modules

SUMMARY


Refer : https://issues.redhat.com/browse/ACA-1476
This PR updates the module documentation with the correct return values for the following modules
cloudtrail, cloudtrail_info, cloudwatchevent_rule, cloudwatchlogs_log_group
ISSUE TYPE


Bugfix Pull Request
Docs Pull Request
Feature Pull Request
New Module Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Alina Buzachis
Reviewed-by: Mike Graves <[email protected]>
…nsible-collections#2083)

backup_plan_info: bugfix to enable getting info of all backup plans

SUMMARY

Fix being unable to fetch info of all backup plans.
With backup_plan_names being a required parameter, the functionality to get all plans info was not working
# Gather information about all backup plans
- name: Get info of all backup plans 
  amazon.aws.backup_plan_info:
  register: plan_info_result

gave
**********
fatal: [localhost]: FAILED! => {"changed": false, "msg": "missing required arguments: backup_plan_names"}



ISSUE TYPE


Bugfix Pull Request

COMPONENT NAME

backup_plan_info
ADDITIONAL INFORMATION

Reviewed-by: GomathiselviS
Reviewed-by: Alina Buzachis
Reviewed-by: Mandar Kulkarni <[email protected]>
Reviewed-by: Mark Chappell
…ttings parameter (ansible-collections#2124)

backup_plan: modify args spec and add examples for advanced_backup_settings parameter

Fixes ansible-collections#2110
This PR

adds an example to showcase the usage of advanced_backup_settings parameter
adds option in args spec and suboption in module doc

SUMMARY


ISSUE TYPE


Docs Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Alina Buzachis
Reviewed-by: Bikouo Aubin
Reviewed-by: Helen Bailey <[email protected]>
Update return block of Backup modules

SUMMARY

Updated return block of following modules, tried to ensure consistency across their return blocks
1. backup_plan
2. backup_plan_info
3. backup_vault
4. backup_selection


ISSUE TYPE


Docs Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Alina Buzachis
Reviewed-by: Mandar Kulkarni <[email protected]>
Reviewed-by: Bikouo Aubin
Reviewed-by: Mike Graves <[email protected]>
…#2131)

Update changelog after release 7.6.1

SUMMARY

Update changelog after release 7.6.1

ISSUE TYPE


Bugfix Pull Request
Docs Pull Request
Feature Pull Request
New Module Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: GomathiselviS
Reviewed-by: Mike Graves <[email protected]>
Code refactoring - module_utils/elbv2

SUMMARY

closes ansible-collections#1981

ISSUE TYPE


Feature Pull Request

Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Bikouo Aubin
Reviewed-by: Alina Buzachis
SUMMARY


This  PR updates the return blocks for lambda modules.
Refer:  https://issues.redhat.com/browse/ACA-1478
ISSUE TYPE


Bugfix Pull Request
Docs Pull Request
Feature Pull Request
New Module Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Alina Buzachis
Reviewed-by: Bikouo Aubin
Reviewed-by: Mike Graves <[email protected]>
Reviewed-by: Mark Chappell
SUMMARY
This is an initial refactor PR for rds modules, focusing on documentation and shared boto3 client functionality in rds module_utils, rds_instance_info, and rds_instance. First PR for ansible-collections#2003 / https://issues.redhat.com/browse/ACA-1343.
COMPONENT NAME
rds_instance_info
rds_instance
module_utils/rds.py
ADDITIONAL INFORMATION
Detailed summary of all the changes:
module_utils/rds.py:

Add RDS error class and handler
Add describe_db_instances(), describe_db_snapshots(), and list_tags_for_resource() functions to handle boto3 client call
Refactor get_tags() to use new list_tags_for_resource() function
Add type hinting and function docstrings

rds_instance_info module:

Replace internal error handler and _describe_db_instances() functions with calls to new functions from module_utils/rds.py
Remove extra boto3 call to retrieve tags for resource and just reformat instance TagList attribute since it is always returned by describe_db_instances
Update instance_info() function return value for clarity
Add type hinting and function docstrings
Remove unit tests for functions no longer in module
Refactor remaining unit tests to match updated instance_info() function

rds_instance module:

Refactor internal get_instance() function to use describe_db_instances() from module_utils/rds.py, remove extra boto3 call to get resource tags, and remove manual retry logic
Refactor internal get_final_snapshot() function to use describe_db_snapshots() from module_utils/rds.py
Add type hinting and function docstrings, and in some cases inline comments to explain complex logic
Add unit tests for refactored functions

Reviewed-by: Alina Buzachis
Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Mandar Kulkarni <[email protected]>
Reviewed-by: Mike Graves <[email protected]>
Reviewed-by: Bikouo Aubin
Reviewed-by: Mark Chappell
…ance* (ansible-collections#2099)

SUMMARY

This pull request contains a set of changes:
module_utils/ec2.py: add a set of AWS API wrapped into retry and AnsibleEC2ErrorHandler, some of them will be used later to refactor others ec2_* modules
Refactorec2_vol*, ec2_nat_gateway*, ec2_vpc_igw*, ec2_snapshot*, ec2_spot_instance* modules: add type hint, the main() function should be limited to call either ensure_[absent|present] functions or any others function to describe resources.

The changes in this PR have been extracted from ansible-collections#2076

ISSUE TYPE


Feature Pull Request

COMPONENT NAME


ec2_snapshot
ec2_snapshot_info
ec2_spot_instance
ec2_spot_instance_info
ec2_vol
ec2_vol_info
ec2_vpc_igw
ec2_vpc_igw_info
ec2_vpc_nat_gateway
ec2_vpc_nat_gateway_info

ADDITIONAL INFORMATION

Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Bikouo Aubin
Reviewed-by: GomathiselviS
SUMMARY


Refer https://issues.redhat.com/browse/ACA-1477
This PR updates the return block in the route53_health_check module documentation.
ISSUE TYPE


Docs Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Bikouo Aubin
Reviewed-by: Mike Graves <[email protected]>
…#2126)

SUMMARY

Update changelog after release 8.0.1

ISSUE TYPE


Bugfix Pull Request
Docs Pull Request
Feature Pull Request
New Module Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: GomathiselviS
Reviewed-by: Mike Graves <[email protected]>
Reviewed-by: Mark Chappell
Reviewed-by: Alina Buzachis
alinabuzachis and others added 13 commits June 24, 2024 18:29
…s the correct formats (ansible-collections#2086)

SUMMARY


ISSUE TYPE


Bugfix Pull Request
Docs Pull Request
Feature Pull Request
New Module Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Mandar Kulkarni <[email protected]>
Reviewed-by: Mark Chappell
…ons#2136)

SUMMARY


Refer https://issues.redhat.com/browse/ACA-1477
Route53 module, returns after and before diff as module output. This PR fixes this bug, by adding a check to return the diff only when module._diff is set to true.
ISSUE TYPE


Bugfix Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Bikouo Aubin
Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Alina Buzachis
…ormation (ansible-collections#2127)

SUMMARY

Updated return block of following modules
aws_region_info
autoscaling_group_info
cloudformation


ISSUE TYPE


Docs Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Alina Buzachis
Reviewed-by: Mandar Kulkarni <[email protected]>
Reviewed-by: GomathiselviS
Reviewed-by: Mike Graves <[email protected]>
SUMMARY

Fix back-ticks in changelogs

ISSUE TYPE


Bugfix Pull Request
Docs Pull Request
Feature Pull Request
New Module Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Mark Chappell
…collections#2097)

SUMMARY

This pull request contains a set of changes:

module_utils/ec2.py: add a set of AWS API wrapped into retry and AnsibleEC2ErrorHandler, some of them will be used later to refactor others ec2_* modules
Refactor ec2_vpc_endpoint* and ec2_vpc_dhcp_option* modules: add type hint, the main() function should be limited to call either ensure_[absent|present] functions or any others function to describe resources.


The changes in this PR have been extracted from ansible-collections#2076
ISSUE TYPE


Feature Pull Request

COMPONENT NAME


ec2_vpc_dhcp_option_info
ec2_vpc_dhcp_option
ec2_vpc_endpoint_info
ec2_vpc_endpoint
ec2_vpc_endpoint_service_info

ADDITIONAL INFORMATION

Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Bikouo Aubin
Reviewed-by: Alina Buzachis
SUMMARY
Amazon's dropping support for M4/R4/T2 RDS DB instances, replace their use in tests before something suddenly fails.
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
rds
ADDITIONAL INFORMATION

Reviewed-by: Alina Buzachis
…ollections#2135)

SUMMARY

Updated return block of following modules
- s3_bucket
- s3_bucket_info
- s3_object


ISSUE TYPE


Docs Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Mandar Kulkarni <[email protected]>
Reviewed-by: Mark Chappell
Reviewed-by: Alina Buzachis
…aurora clusters (ansible-collections#2063)

SUMMARY

Adds support for I/O-Optimized storage configuration for aurora clusters

Fixes ansible-collections#2038
ISSUE TYPE


Feature Pull Request

COMPONENT NAME

rds_cluster
ADDITIONAL INFORMATION

Reviewed-by: Alina Buzachis
Reviewed-by: Mandar Kulkarni <[email protected]>
Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: GomathiselviS
SUMMARY

Updated return block of following modules
- kms_key
- kms_key_info


ISSUE TYPE


Docs Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Alina Buzachis
Reviewed-by: GomathiselviS
…s the correct formats (ansible-collections#2088)

SUMMARY

Ensures the modules documentation follows the correct formats

ISSUE TYPE


Bugfix Pull Request
Docs Pull Request
Feature Pull Request
New Module Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: GomathiselviS
…s the correct formats (ansible-collections#2087)

SUMMARY

Ensures the modules documentation follows the correct formats

ISSUE TYPE


Bugfix Pull Request
Docs Pull Request
Feature Pull Request
New Module Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Mandar Kulkarni <[email protected]>
Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: GomathiselviS
Reviewed-by: Alina Buzachis
Signed-off-by: Alina Buzachis <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants