Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support AWS_MSK_IAM authentication #147

Closed
wants to merge 5 commits into from

Conversation

wbarnha
Copy link
Owner

@wbarnha wbarnha commented Mar 8, 2024

This pull request addresses issue dpkp#2232 by adding an AWS_MSK_IAM authentication mechanism.
A detailed description of the authentication scheme is available here:

I understand that kafka-python may not be the appropriate place to put a vendor-specific authentication mechanism.
If that's the case maybe it's better suited as a plug-in?
The library doesn't support auth extensions at the moment but it doesn't look like a huge lift to get there.

To use the mechanism pass the following keyword arguments when
initializing a class:

security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM',
bootstrap_servers=[
    'b-1.cluster.x.y.kafka.region.amazonaws.com:9098',
    ...
],

The credentials and region will be pulled using botocore.session.Session.
Using the mechanism requires the botocore library which can be
installed with:

pip install botocore

TODO:

  • Documentation
  • Test authentication payload generation
  • Test config verification / auth method
  • Refresh mechanism for temporary credentials?

This change is Reviewable

mattoberle and others added 5 commits August 18, 2021 16:22
Adds an AWS_MSK_IAM authentication mechanism which is described here:
* https://github.com/aws/aws-msk-iam-auth#uriencode

To use the mechanism pass the following keyword arguments when
initializing a class:

```
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM',
bootstrap_servers=[
    'b-1.cluster.x.y.kafka.region.amazonaws.com:9088',
    ...
],
```

The credentials and region will be pulled using `botocore.session.Session`.
Using the mechanism requires the `botocore` library which can be
installed with:

```sh
pip install botocore
```

**TODO:**

- [ ] Documentation
- [ ] Tests
- [ ] Refresh mechanism for temporary credentials?
The two tests in `test/test_msk.py` should ensure that the changes to
`kafka/msk.py` do not break the authentication payload.

The authentication payload was validated using a real AWS Kafka cluster
before adding tests with the hard-coded signatures.
@wbarnha
Copy link
Owner Author

wbarnha commented Mar 18, 2024

Resuming in #170.

@wbarnha wbarnha closed this Mar 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants