-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Will this library be released for other languages, like .net or nodejs? #10
Comments
plus "python" 🙂 |
Hi thanks for your comment and interest in using IAM access control for MSK. Many of the Kafka clients for non-JVM languages use a C-based library called librdkafka. This will be the place that this authentication mechanism needs to be supported. From an AWS perspective, it would be great to hear about any other specific language support you might need there. |
There now seems to be an ongoing discussion about this in the librdkafka community. See confluentinc/librdkafka#3385. |
A discussion has also been opened regarding KafkaJS, which is not binding to librdkafka. See tulios/kafkajs#1098. |
Hi All, Our dataflow looks like this [DMS, Lambda(python)] -> MSK -> EMR Spark (python). On python there are two main kafka packages:
On kafka-python I have started discussion. See dpkp/kafka-python#2232 (update): |
For potential cross-language implementation reference, I've added support for AWS_MSK_IAM in my franz-go client. I've tested it against a scrappy cluster setup, but I'd be happy to have broader testing (and I'm planning to turn down my cluster imminently). |
I would be really helpful if there was a specification somewhere explaining how this authentication mechanism is supposed to work, so that each library author doesn't have to reverse engineer the Java implementation. @twmb's Go implementation is at least easier to read, but it would be preferable to not have to rely on a game of telephone. EDIT: @twmb could you share how you managed to work against the MSK cluster? I'm getting what looks like a nonsense response to the ApiVersions request and then the connection is immediately closed, so I don't even get to the authentication flow.
|
@Nevon it is necessary to use TLS encryption when working with IAM access control. Plaintext channels are not supported. Meanwhile, we are in the process of improving the documentation of the SASL mechanism, so it becomes easier to implement in other languages. |
Ah, thank you! 🤦♂️ Now I'm getting something that looks like it makes more sense. |
@Nevon fwiw I usually use my kcl command line program to test things, and I added support for edit1: also, I recommend using the latest commit, which has a slight change in how the configuration is loaded |
We have updated the README to add details on how the authentication payload is generated. It would be really great if you could let us know if this helps. |
Thanks for adding the additional information to the readme. I was working on an initial implementation using @aws-sdk/signature-v4 yesterday, but my request was rejected, so I think I'll need to do some wire capture to see where I'm going wrong. Hopefully I'll have some time to go back to it this weekend. |
@Nevon Please let us know if we can help by looking at some code, even if it is an early version. |
Haven't had any time to work on this yet, as some personal stuff had to take priority. If anyone else wants to take a crack at it and provide a third-party plugin, definitely don't let me stop you. There's no telling if I'll have the time to finish this. I created tulios/kafkajs#1101 to allow for injecting custom authentication mechanisms at runtime. I've been building the msk iam auth plugin using this interface. There are some things I'm not super happy with in terms of the typings, so maybe there will be some changes there to make it easier to work with, but overall that's how you'd extend KafkaJS with a new authentication mechanism. |
I managed to connect to MSK with IAM with Node.js and kafkajs by not using @AWS-SDK.
|
The Node.js aws sdk supports credential providers which can extract temporary credentials.
Please check if this will serve your purpose:
https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/modules/_aws_sdk_credential_provider_node.html
It describes how the credential provider can extract credentials from an instance role or container role or assume another role.
On another note, are you planning to contribute this integration to kafkajs?
Sayantan
On Jul 9, 2021, at 5:15 PM, Jaka Maver ***@***.***> wrote:
I managed to connect to MSK with IAM with Node.js and kafkajs by not using @AWS-SDK.
Instead I manually wrote all the code required for generating authentication payload, signature, canonical request etc.
The part where I am having trouble is with generating temporary credentials (access key, secret key, token).
Currently I am using the following way to generate temporary credentials: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials
Does Node.js @AWS-SDK package have a way to generate (and retrieve) temporary credentials?
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \
&& curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<#10 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIA3ZZRXQPRD7TW3YRDYOHTTW6GHVANCNFSM44IFMBGA>.
|
Thank you. I will check that out. |
@sayantacC Hey, can you explain to me what does EXPIRES_IN header mean? |
The
In cases 1 and 2 above, the client will need to reauthenticate in order to continue making any further Kafka API requests over the same connection. If the client reauthenticates successfully, then it can continue using the same connection, else the broker will terminate the connection. |
Hi , |
Is this still true? The people at librdkafka say that this is something that they will not implement and is the responsibility of AWS. confluentinc/librdkafka#3780 (comment) If it is, we then have a stalemate and really important functionality won't be implemented |
Please create a Python package to support MSK Serverless (don't rely on third parties to do this when it seems they won't!) |
We're a .NET Core shop that is using MSK with IAM and the Confluent Kafka library. Any news on a .NET Core plugin for IAM or is the solution to move the cluster to a different auth framework? |
First of all: +1 on this issue, since I would also love to see a release of this library for C# and NodeJS. As there seems to be no progress on this for 1.5 years I do have another question related to it. As long as there is no support for C# and/or NodeJS, we thought of setting up a Java Kafka proxy that uses the
I don't see a clear way forward with this approach either. Is there another kafkaproxy someone knows of that does actually support the |
+1 for .NET support |
kafka-python + SASL MSK worker for me
|
@simonkarman Here is an example of Confluent's Kafka Rest Proxy connecting with MSK using the SASL/IAM endpoint. I know the proxy will start up and connect with MSK; however, I have not extensively tested the proxy to ensure it is fully functioning. |
@grsubramanian Are you aware of this thread in the librdkafka. It seems like AWS should step up and use its vast resources to contribute back to the open source project. Especially when you consider the open-source community would need to run potentially costly MSK servers to dev/test. Otherwise, I am afraid we are in a stalemate and nothing will get done. |
@sayantacC is currently the best POC on this. |
did anyone got a solution for .Net application ? @robertbarron |
I don't think this gives the full functionality. From here:
|
Hi All, We are happy to share that you can now use AWS IAM authN/Z for your MSK clusters using Amazon MSK's open-sourced client helper libraries for various languages. The new feature allows customers to send IAM sigv4 signatures via SASL/OAUTHBEARER, an open standard for authorization and authentication. For more details and getting started, check the README sections of the client libraries. Python: https://github.com/aws/aws-msk-iam-sasl-signer-python This feature is now available to all existing MSK Serverless and Provisioned clusters. |
|
Is this also implemented for C/C++? |
Hi @NaodFessahaye we have same needs for C/C++, opened another feature request at #186 |
Hello,
We've been using a production level Amazon MSK for about a year now and saw the news about IAM integration. Our company has a few hundred microservices that connect to Amazon MSK via TLS. These microservices are build on .NET or Nodejs.
So, the main question is if you guys are planning on releasing this library for .NET, nodeJS or other languages.
Thanks
The text was updated successfully, but these errors were encountered: