feat(kafka): aws_msk_config and aws_msk_cluster modules have been added #547

oukooveu · 2021-04-18T09:21:21Z

SUMMARY

Modules for Amazon Managed Streaming for Apache Kafka (Amazon MSK) have been added. There are two modules: aws_msk_config for managing MSK configuration and aws_msk_cluster -- for clusters.

ISSUE TYPE

New Module Pull Request

COMPONENT NAME

aws_msk_config
aws_msk_cluster

ADDITIONAL INFORMATION

All MSK cluster related operations are time consuming, for example create takes 20-30 minutes, update Kafka version -- more than one hour, update configuration -- 10-15 minutes.

Integration tests work in my own account, please let me know if I can help with an evidences.

REFERENCES

Related aws-terminator PR: mattclay/aws-terminator#143
Boto3: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kafka.html

ansibullbot · 2021-04-18T09:32:19Z

cc @jillr @s-hertel @tremble @wimnat
click here for bot help

tremble

I've not had chance to go through this 100%

Here's some initial feedback.

plugins/modules/aws_msk_config.py

tests/integration/targets/aws_msk_config/aliases

tremble · 2021-04-19T12:11:28Z

Thanks for taking the time to open this PR. If you need help (or have more questions about our CI) feel free to reach out on #ansible-aws again.

jillr · 2021-04-22T22:01:24Z

MSK creation and updates are quite slow, the test suite took around 30 minutes to run for me, we might want to mark these tests unstable so they only run when there are changes to the modules.

jillr · 2021-05-04T22:14:54Z

recheck

alinabuzachis · 2021-05-17T09:19:48Z

recheck

alinabuzachis · 2021-05-20T15:10:07Z

recheck

ansibullbot · 2021-05-25T15:26:52Z

@oukooveu this PR contains the following merge commits:

71148d8

Please rebase your branch to remove these commits.

click here for bot help

alinabuzachis · 2021-05-25T15:29:57Z

@oukooveu Thank you for working on these new modules. Could you please rebase your PR?

oukooveu · 2021-05-25T15:32:08Z

@oukooveu Thank you for working on these new modules. Could you please rebase your PR?

done

oukooveu · 2021-06-23T08:51:32Z

If the tests can be run in the CI, please remove the unsupported flag.

I'm not sure required policies were applied to target account, waiting for @jillr for confirmation.

jillr · 2021-06-24T22:16:22Z

@oukooveu Yes, your terminator PR was merged and the new policies were deployed to our CI account

oukooveu · 2021-06-26T09:36:40Z

@jillr I've removed unsupported from aliases and tests were passed successfully but actually integration tests were not started and ansible-test: Run the test suite task was skipped in all three ansible-test-cloud-integration-aws-py36 batches. Don't understand why, the condition is:

- name: Run the test suite
  args:
    chdir: "{{ _test_location }}"
    executable: /bin/bash
  environment: "{{ ansible_test_environment | default({}) }}"
  shell: "source {{ ansible_test_venv_path }}/bin/activate; {{ ansible_test_executable }} {{ ansible_test_test_command }} {{ ansible_test_options }} {{ ansible_test_verbosity }} {{ _integration_targets }}"
  when: (ansible_test_changed|bool == False) or (ansible_test_changed|bool and _integration_targets|length > 0)

and _integration_targets contains changed modules as we can see in job output:

also ansible_test_changed should be true because of definition in zuul.d/ansible-cloud-jobs.yaml:

- job:
    name: ansible-test-cloud-integration-aws-py36
    parent: ansible-core-ci-aws-session
    nodeset: controller-python36
    dependencies:
      - name: build-ansible-collection
    pre-run:
      - playbooks/ansible-test-base/pre.yaml
      - playbooks/ansible-cloud/aws/pre.yaml
    run: playbooks/ansible-test-base/run.yaml
    # TODO: disabled for now, required for Ara
    # post-run: playbooks/ansible-test-base/post.yaml
    required-projects:
      - name: github.com/ansible/ansible
    timeout: 3600
    vars:
      ansible_test_collections: true
      ansible_collections_repo: "{{ zuul.project.canonical_name }}"
      ansible_test_command: integration
      ansible_test_enable_ara: false
      ansible_test_python: 3.6
      ansible_test_changed: true
      ansible_test_split_in: 3

Could you or @goneri take a look please?

oukooveu · 2021-06-26T10:58:04Z

I've found what's wrong with tests run, they are splitting in incorrect way, the issue is in roles/ansible-test/tasks/split_targets.yaml:

- set_fact:
    _start_at: "{{ number_entries|int // ansible_test_split_in * (ansible_test_do_number -1)|int }}"
    _end_at: "{{ (number_entries|int // ansible_test_split_in * (ansible_test_do_number |int)) }}"

and when ansible_test_split_in is greater than number_entries we get empty list.

Possible fix (if you need even distribution): ansible/ansible-zuul-jobs#968

oukooveu · 2021-06-26T14:02:42Z

aws_msk_config tests are okay, aws_msk_cluster tests are failed with:

2021-06-26 13:48:38.225976 | centos-8 | TASK [aws_msk_cluster : create msk cluster (check mode)] ***********************
2021-06-26 13:48:38.226082 | centos-8 | task path: /home/zuul/.ansible/collections/ansible_collections/community/aws/tests/integration/targets/aws_msk_cluster/tasks/test_create.yml:2
2021-06-26 13:48:38.258133 | centos-8 | <testhost> ESTABLISH LOCAL CONNECTION FOR USER: zuul
2021-06-26 13:48:38.258388 | centos-8 | <testhost> EXEC /bin/sh -c 'echo ~zuul && sleep 0'
2021-06-26 13:48:38.263996 | centos-8 | <testhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/zuul/.ansible/tmp `"&& mkdir "` echo /home/zuul/.ansible/tmp/ansible-tmp-1624715318.2630663-4289-87016496591224 `" && echo ansible-tmp-1624715318.2630663-4289-87016496591224="` echo /home/zuul/.ansible/tmp/ansible-tmp-1624715318.2630663-4289-87016496591224 `" ) && sleep 0'
2021-06-26 13:48:38.398708 | centos-8 | Using module file /home/zuul/.ansible/collections/ansible_collections/community/aws/plugins/modules/aws_msk_cluster.py
2021-06-26 13:48:38.399639 | centos-8 | <testhost> PUT /home/zuul/.ansible/tmp/ansible-local-4087ta6jc607/tmpd7mtwhki TO /home/zuul/.ansible/tmp/ansible-tmp-1624715318.2630663-4289-87016496591224/AnsiballZ_aws_msk_cluster.py
2021-06-26 13:48:38.400139 | centos-8 | <testhost> EXEC /bin/sh -c 'chmod u+x /home/zuul/.ansible/tmp/ansible-tmp-1624715318.2630663-4289-87016496591224/ /home/zuul/.ansible/tmp/ansible-tmp-1624715318.2630663-4289-87016496591224/AnsiballZ_aws_msk_cluster.py && sleep 0'
2021-06-26 13:48:38.406887 | centos-8 | <testhost> EXEC /bin/sh -c 'ANSIBLE_DEBUG_BOTOCORE_LOGS=True /home/zuul/venv/bin/python3.6 /home/zuul/.ansible/tmp/ansible-tmp-1624715318.2630663-4289-87016496591224/AnsiballZ_aws_msk_cluster.py && sleep 0'
2021-06-26 13:48:39.092279 | centos-8 | <testhost> EXEC /bin/sh -c 'rm -f -r /home/zuul/.ansible/tmp/ansible-tmp-1624715318.2630663-4289-87016496591224/ > /dev/null 2>&1 && sleep 0'
2021-06-26 13:48:39.101776 | centos-8 | The full traceback is:
2021-06-26 13:48:39.101799 | centos-8 | Traceback (most recent call last):
2021-06-26 13:48:39.101807 | centos-8 |   File "/tmp/ansible_aws_msk_cluster_payload_5j4bpvgy/ansible_aws_msk_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/aws_msk_cluster.py", line 298, in find_cluster_by_name
2021-06-26 13:48:39.101814 | centos-8 |   File "/tmp/ansible_aws_msk_cluster_payload_5j4bpvgy/ansible_aws_msk_cluster_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 154, in retry_func
2021-06-26 13:48:39.101821 | centos-8 |     raise e
2021-06-26 13:48:39.101828 | centos-8 |   File "/tmp/ansible_aws_msk_cluster_payload_5j4bpvgy/ansible_aws_msk_cluster_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 144, in retry_func
2021-06-26 13:48:39.101834 | centos-8 |     return f(*args, **kwargs)
2021-06-26 13:48:39.101840 | centos-8 |   File "/tmp/ansible_aws_msk_cluster_payload_5j4bpvgy/ansible_aws_msk_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/aws_msk_cluster.py", line 287, in list_clusters_with_backoff
2021-06-26 13:48:39.101846 | centos-8 |   File "/home/zuul/venv/lib/python3.6/site-packages/botocore/paginate.py", line 449, in build_full_result
2021-06-26 13:48:39.101853 | centos-8 |     for response in self:
2021-06-26 13:48:39.101859 | centos-8 |   File "/home/zuul/venv/lib/python3.6/site-packages/botocore/paginate.py", line 255, in __iter__
2021-06-26 13:48:39.101865 | centos-8 |     response = self._make_request(current_kwargs)
2021-06-26 13:48:39.101872 | centos-8 |   File "/home/zuul/venv/lib/python3.6/site-packages/botocore/paginate.py", line 332, in _make_request
2021-06-26 13:48:39.101878 | centos-8 |     return self._method(**current_kwargs)
2021-06-26 13:48:39.101886 | centos-8 |   File "/home/zuul/venv/lib/python3.6/site-packages/botocore/client.py", line 386, in _api_call
2021-06-26 13:48:39.101900 | centos-8 |     return self._make_api_call(operation_name, kwargs)
2021-06-26 13:48:39.101906 | centos-8 |   File "/home/zuul/venv/lib/python3.6/site-packages/botocore/client.py", line 705, in _make_api_call
2021-06-26 13:48:39.101911 | centos-8 |     raise error_class(parsed_response, operation_name)
2021-06-26 13:48:39.101917 | centos-8 | botocore.errorfactory.BadRequestException: An error occurred (BadRequestException) when calling the ListClusters operation: The resource name contains too many or too few characters

alinabuzachis

@oukooveu Thank you for working on this. LGTM.

goneri · 2021-07-02T21:30:21Z

Thank you @oukooveu!

feat(kafka): aws_msk_config and aws_msk_cluster modules have been added SUMMARY Modules for Amazon Managed Streaming for Apache Kafka (Amazon MSK) have been added. There are two modules: aws_msk_config for managing MSK configuration and aws_msk_cluster -- for clusters. ISSUE TYPE New Module Pull Request COMPONENT NAME aws_msk_config aws_msk_cluster ADDITIONAL INFORMATION All MSK cluster related operations are time consuming, for example create takes 20-30 minutes, update Kafka version -- more than one hour, update configuration -- 10-15 minutes. Integration tests work in my own account, please let me know if I can help with an evidences. REFERENCES Related aws-terminator PR: mattclay/aws-terminator#143 Boto3: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kafka.html Reviewed-by: Mark Chappell <None> Reviewed-by: Daniil Kupchenko <None> Reviewed-by: Alina Buzachis <None> Reviewed-by: Gonéri Le Bouder <[email protected]> This commit was initially merged in https://github.com/ansible-collections/community.aws See: ansible-collections@3bfcfe3

feat(kafka): aws_msk_config and aws_msk_cluster modules have been added SUMMARY Modules for Amazon Managed Streaming for Apache Kafka (Amazon MSK) have been added. There are two modules: aws_msk_config for managing MSK configuration and aws_msk_cluster -- for clusters. ISSUE TYPE New Module Pull Request COMPONENT NAME aws_msk_config aws_msk_cluster ADDITIONAL INFORMATION All MSK cluster related operations are time consuming, for example create takes 20-30 minutes, update Kafka version -- more than one hour, update configuration -- 10-15 minutes. Integration tests work in my own account, please let me know if I can help with an evidences. REFERENCES Related aws-terminator PR: mattclay/aws-terminator#143 Boto3: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kafka.html Reviewed-by: Mark Chappell <None> Reviewed-by: Daniil Kupchenko <None> Reviewed-by: Alina Buzachis <None> Reviewed-by: Gonéri Le Bouder <[email protected]> This commit was initially merged in https://github.com/ansible-collections/community.aws See: ansible-collections@7d39179

feat(kafka): aws_msk_config and aws_msk_cluster modules have been added SUMMARY Modules for Amazon Managed Streaming for Apache Kafka (Amazon MSK) have been added. There are two modules: aws_msk_config for managing MSK configuration and aws_msk_cluster -- for clusters. ISSUE TYPE New Module Pull Request COMPONENT NAME aws_msk_config aws_msk_cluster ADDITIONAL INFORMATION All MSK cluster related operations are time consuming, for example create takes 20-30 minutes, update Kafka version -- more than one hour, update configuration -- 10-15 minutes. Integration tests work in my own account, please let me know if I can help with an evidences. REFERENCES Related aws-terminator PR: mattclay/aws-terminator#143 Boto3: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kafka.html Reviewed-by: Mark Chappell <None> Reviewed-by: Daniil Kupchenko <None> Reviewed-by: Alina Buzachis <None> Reviewed-by: Gonéri Le Bouder <[email protected]>

…ns#548) ec2_key - add support for tagging and returning id SUMMARY Add support for tagging key pairs and also return the key ID ISSUE TYPE Feature Pull Request COMPONENT NAME ec2_key ADDITIONAL INFORMATION fixes: ansible-collections#546 fixes: ansible-collections#547 Reviewed-by: Alina Buzachis <None> Reviewed-by: None <None>

ansibullbot added community_review integration tests/integration module module needs_triage new_contributor Help guide this first time contributor new_module New module new_plugin New plugin plugins plugin (any type) tests tests labels Apr 18, 2021

oukooveu marked this pull request as ready for review April 18, 2021 12:10

oukooveu mentioned this pull request Apr 18, 2021

Support for AWS MSK services has been added mattclay/aws-terminator#143

Merged

tremble requested changes Apr 19, 2021

View reviewed changes

ansibullbot removed the needs_triage label Apr 19, 2021

oukooveu marked this pull request as draft April 19, 2021 18:58

oukooveu marked this pull request as ready for review April 20, 2021 06:56

oukooveu requested a review from tremble April 20, 2021 06:57

ansibullbot added merge_commit This PR contains at least one merge commit. Please resolve! needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html and removed community_review labels May 25, 2021

ansibullbot removed merge_commit This PR contains at least one merge commit. Please resolve! needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html labels May 25, 2021

oukooveu added 2 commits June 25, 2021 17:04

chore: enable tests for aws_msk_config module

f9538e8

chore: enable tests for aws_msk_cluster module

9f112d6

oukooveu closed this Jun 26, 2021

oukooveu reopened this Jun 26, 2021

oukooveu added 2 commits June 27, 2021 10:02

fix: cluster name should not exceed 64 characters limit

baac319

chore: shippable groups have been removed from aliases

3fa72b8

oukooveu closed this Jun 29, 2021

oukooveu reopened this Jun 29, 2021

oukooveu mentioned this pull request Jun 29, 2021

fix: correct computation for range end has been added ansible/ansible-zuul-jobs#975

Merged

oukooveu closed this Jun 29, 2021

oukooveu reopened this Jun 29, 2021

oukooveu closed this Jun 29, 2021

oukooveu reopened this Jun 29, 2021

alinabuzachis approved these changes Jun 30, 2021

View reviewed changes

goneri approved these changes Jul 2, 2021

View reviewed changes

goneri added the gate label Jul 2, 2021

ansible-zuul bot merged commit 3bfcfe3 into ansible-collections:main Jul 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(kafka): aws_msk_config and aws_msk_cluster modules have been added #547

feat(kafka): aws_msk_config and aws_msk_cluster modules have been added #547

oukooveu commented Apr 18, 2021 •

edited

Loading

ansibullbot commented Apr 18, 2021

tremble left a comment

tremble commented Apr 19, 2021

jillr commented Apr 22, 2021

jillr commented May 4, 2021

alinabuzachis commented May 17, 2021

alinabuzachis commented May 20, 2021

ansibullbot commented May 25, 2021

alinabuzachis commented May 25, 2021

oukooveu commented May 25, 2021

oukooveu commented Jun 23, 2021

jillr commented Jun 24, 2021

oukooveu commented Jun 26, 2021

oukooveu commented Jun 26, 2021 •

edited

Loading

oukooveu commented Jun 26, 2021

alinabuzachis left a comment

goneri commented Jul 2, 2021

feat(kafka): aws_msk_config and aws_msk_cluster modules have been added #547

feat(kafka): aws_msk_config and aws_msk_cluster modules have been added #547

Conversation

oukooveu commented Apr 18, 2021 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

REFERENCES

ansibullbot commented Apr 18, 2021

tremble left a comment

Choose a reason for hiding this comment

tremble commented Apr 19, 2021

jillr commented Apr 22, 2021

jillr commented May 4, 2021

alinabuzachis commented May 17, 2021

alinabuzachis commented May 20, 2021

ansibullbot commented May 25, 2021

alinabuzachis commented May 25, 2021

oukooveu commented May 25, 2021

oukooveu commented Jun 23, 2021

jillr commented Jun 24, 2021

oukooveu commented Jun 26, 2021

oukooveu commented Jun 26, 2021 • edited Loading

oukooveu commented Jun 26, 2021

alinabuzachis left a comment

Choose a reason for hiding this comment

goneri commented Jul 2, 2021

oukooveu commented Apr 18, 2021 •

edited

Loading

oukooveu commented Jun 26, 2021 •

edited

Loading