amazon.aws.aws_secret lookup with bypath=true returns only upto 10 values

[ggrzesiuk]

Summary

I found an issue when using ansible lookup plugin. In my ansible playbook ( Ansible 2.10) I'm using lookup plugin to get all secrets from AWS Secrets Manager with the bypath attribute.

vars: 
#which environment: dev, test,uat, prd 
my_env: dev 
aws_secret_path: mypath/{{my_env}}

name: "get all secrets from AWS Secrets Manager" 
  set_fact: secret_value: "{{ lookup('amazon.aws.aws_secret', '{{aws_secret_path}}', on_missing='skip', bypath='true', region='eu-west-1' )}}"

It is working fine however I realized that it returns only up to 10 elements.
Is there a way to return all elements or use pagination ? Have you faced similar issue with lookup plugin ?

Issue Type

Bug Report

Component Name

amazon.aws.aws_secret

Ansible Version

 ansible --version
ansible 2.10.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /venv/awx-venv/lib64/python3.6/site-packages/ansible
  executable location = /venv/awx-venv/bin/ansible
  python version = 3.6.8 (default, Apr 16 2020, 01:36:27) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]

Collection Versions

Collection        Version
----------------- -------
amazon.aws        1.5.0  
community.aws     1.4.0  
community.general 2.3.0 

AWS SDK versions

(awx-venv) bash-4.4$ pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: [email protected]
License: MIT
Location: /venv/awx-venv/lib/python3.6/site-packages
Requires: 
Required-by: 
---
Name: boto3
Version: 1.17.46
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /venv/awx-venv/lib/python3.6/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.20.46
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /venv/awx-venv/lib/python3.6/site-packages
Requires: python-dateutil, jmespath, urllib3
Required-by: s3transfer, boto3

Configuration

$ ansible-config dump --only-changed

OS / Environment

No response

Steps to Reproduce

vars: 
#which environment: dev, test,uat, prd 
my_env: dev 
aws_secret_path: mypath/{{my_env}}

name: "get all secrets from AWS Secrets Manager" 
  set_fact: secret_value: "{{ lookup('amazon.aws.aws_secret', '{{aws_secret_path}}', on_missing='skip', bypath='true', region='eu-west-1' )}}"

Expected Results

Lookup plugin should return all secrets from AWS Secret Manager

Actual Results

It returns only up tp 10 results

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:
None

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ggrzesiuk]

As sugested by mdaniel on Stackoverflow It seems you are ignoring the advice to honor NextToken -
as described here Kindly please implement it as it doesn't allow to fetch more than 10 records from AWS Secrets Manager

[ggrzesiuk]

hi! any update on this issue ? Would good if that could be fixed.
Thanks!
Greg

[mandar242]

hi! any update on this issue ? Would good if that could be fixed. Thanks! Greg

Hi Greg, can you please try with this patch #591?

[ggrzesiuk]

Hi Mandar, Thank you for letting me know. Would you be so kind and share with me some info on how can I test that ? I'm not so familiar with ansible galaxy and I'm not sure how can I obtain the release containing this patch and how can I use it. Kindly please let me know. Thank you and best regards, Greg From: Mandar Kulkarni ***@***.***> Sent: Monday, January 17, 2022 10:56 AM To: ansible-collections/amazon.aws ***@***.***> Cc: Grzesiuk, Grzegorz /PL ***@***.***>; Author ***@***.***> Subject: Re: [ansible-collections/amazon.aws] amazon.aws.aws_secret lookup with bypath=true returns only upto 10 values (#472) hi! any update on this issue ? Would good if that could be fixed. Thanks! Greg Hi Greg, can you please try with this patch #591<#591?>? - Reply to this email directly, view it on GitHub<#472 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AU3XZPEITG2IWOJZCLVVSRLUWPRSFANCNFSM5CZECBKQ>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>> The information contained in the e-mail or attached document is CONFIDENTIAL and is intended only for the use of the individual to whom it is addressed. If you are not the intended recipient or employee responsible to deliver to the intended recipient, you are hereby notified that any dissemination, distribution or copying of the communication is strictly prohibited. If you have received the e-mail in error, please immediately notify us and return the original message to us at the address above or destroy it

…

________________________________ Zawarte w niniejszej wiadomo?ci lub za??czonym dokumencie informacje s? POUFNE i przeznaczone tylko dla oznaczonego adresata. Rozpowszechnianie, ujawnianie i kopiowanie tych informacji jest zabronione. Je?eli niniejsza wiadomo?? trafi?a do Pa?stwa przez pomy?k?, bardzo prosimy o powiadomienie nas o tym fakcie i odes?anie dokumentu lub natychmiastowe jego zniszczenie

________________________________ Sanofi-Aventis Sp. z o.o. z siedzib? w Warszawie, przy ul. Bonifraterskiej 17 , 00-203 Warszawa, wpisana do rejestru przedsi?biorc?w Krajowego Rejestru S?dowego prowadzonego przez S?d Rejonowy dla miasta sto?ecznego Warszawy , XII Wydzia? Gospodarczy Krajowego Rejestru S?dowego, pod numerem KRS 0000036286, NIP 813-01-40-525, kapita? zak?adowy w wysoko?ci 50 738 386,94z?.