Skip to content

Commit

Permalink
Document Kafka MSK authentication parameters
Browse files Browse the repository at this point in the history
Signed-off-by: Adrien Fillon <[email protected]>
  • Loading branch information
adrien-f committed Apr 25, 2024
1 parent bcebb00 commit 7089776
Showing 1 changed file with 79 additions and 1 deletion.
80 changes: 79 additions & 1 deletion content/docs/2.14/scalers/apache-kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,10 @@ partition will be scaled to zero. See the [discussion](https://github.com/kedaco
- `version` - Version of your Kafka brokers. See [samara](https://github.com/Shopify/sarama) version (Default: `1.0.0`, Optional)
- `partitionLimitation` - Comma separated list of partition ids to scope the scaling on. Allowed patterns are "x,y" and/or ranges "x-y". If set, the calculation of the lag will only take these ids into account. (Default: All partitions, Optional)
- `sasl` - Kafka SASL auth mode. (Values: `plaintext`, `scram_sha256`, `scram_sha512`, `gssapi`, `oauthbearer`, or `none`, Default: `none`, Optional). This parameter could also be specified in `sasl` in TriggerAuthentication
- `saslTokenProvider` - Kafka SASL token provider when `sasl` is `oauthbearer`. (Values: `bearer`, `aws_msk_iam`, Default: `bearer`, Optional). This parameter could also be specified in `saslTokenProvider` in TriggerAuthentication
- `tls` - To enable SSL auth for Kafka, set this to `enable`. If not set, TLS for Kafka is not used. (Values: `enable`, `disable`, Default: `disable`, Optional). This parameter could also be specified in `tls` in TriggerAuthentication
- `unsafeSsl` - Skip certificate validation when connecting over HTTPS. (Values: `true`, `false`, Default: `false`, Optional)
- `awsRegion` - AWS region of your MSK cluster. (Optional, required for AWS MSK IAM authentication)

> **Note:**
>
Expand All @@ -82,13 +84,15 @@ partition will be scaled to zero. See the [discussion](https://github.com/kedaco

You can use `TriggerAuthentication` CRD to configure the authentication by providing `sasl`, `username` and `password`, in case your Kafka cluster has SASL authentication turned on. If you are using SASL/GSSAPI, you will need to provide Kerberos user, password or keytab, realm and krb5.conf file. If you are using SASL/OAuthbearer you will need to provide `oauthTokenEndpointUri` and `scopes` as required by your OAuth2 provider. You can also add custom SASL extension for OAuthbearer (see [KIP-342](https://cwiki.apache.org/confluence/display/KAFKA/KIP-342%3A+Add+support+for+Custom+SASL+extensions+in+OAuthBearer+authentication)) using `oauthExtensions`.
If TLS is required you should set `tls` to `enable`. If required for your Kafka configuration, you may also provide a `ca`, `cert`, `key` and `keyPassword`. `cert` and `key` must be specified together.
Another alternative is to specify `tls` and `sasl` in ScaledObject instead of `tls` and `sasl` in TriggerAuthentication, respectively.
Another alternative is to specify `tls` and `sasl` in ScaledObject instead of `tls` and `sasl` in TriggerAuthentication, respectively. For AWS MSK IAM authentication, you only need to set `awsRegion` in ScaledObject and you also need to enable TLS by setting `tls` to enable.


**Credential based authentication:**

**SASL:**

- `sasl` - Kafka SASL auth mode. (Values: `plaintext`, `scram_sha256`, `scram_sha512`, `gssapi`, `oauthbearer` or `none`, Default: `none`, Optional)
- `saslTokenProvider` - Kafka SASL token provider. (Values: `bearer`, `aws_msk_iam`, Default: `bearer`, Optional).
- `username` - Username used for sasl authentication. (Optional)
- `password` - Password used for sasl authentication. (Optional)
- `keytab` - Kerberos keytab. Either `password` or `keytab` is required in case of `gssapi`. (Optional)
Expand All @@ -107,6 +111,19 @@ partition will be scaled to zero. See the [discussion](https://github.com/kedaco
- `key` - Key for client authentication. (Optional)
- `keyPassword` - If set the `keyPassword` is used to decrypt the provided `key`. (Optional)

**AWS MSK IAM Specific Configuration:**

For authentication, you must use `TriggerAuthentication` CRD to configure the authenticate by providing `awsAccessKeyID` and `awsSecretAccessKey` or `awsRoleArn` or a pod identity configuration.

**Role based authentication:**

- `awsRoleArn` - Amazon Resource Names (ARNs) uniquely identify AWS resource.

**Credential based authentication:**

- `awsAccessKeyID` - Id of the user.
- `awsSecretAccessKey` - Access key for the user to authenticate with.

### New Consumers and Offset Reset Policy

When a new Kafka consumer is created, it must determine its consumer group initial position, i.e. the offset it will start to read from. The position is decided in Kafka consumers via a parameter `auto.offset.reset` and the possible values to set are `latest` (Kafka default), and `earliest`. This parameter in KEDA should be set accordingly. In this initial status, no offset has been committed to Kafka for the consumer group and any request for offset metadata will return an `INVALID_OFFSET`; so KEDA has to manage the consumer pod's autoscaling in relation to the offset reset policy that has been specified in the parameters:
Expand Down Expand Up @@ -524,3 +541,64 @@ spec:
authenticationRef:
name: keda-trigger-auth-kafka-credential
```

#### Your AWS MSK has IAM auth:

```yaml
apiVersion: v1
kind: Secret
metadata:
name: keda-kafka-secrets
namespace: default
data:
sasl: "oauthbearer"
saslTokenProvider: "aws_msk_iam"
tls: "enable"
awsAccessKeyID: <your awsAccessKeyID>
awsSecretAccessKey: <your awsSecretAccessKey>
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: keda-trigger-auth-kafka-credential
namespace: default
spec:
secretTargetRef:
- parameter: sasl
name: keda-kafka-secrets
key: sasl
- parameter: saslTokenProvider
name: keda-kafka-secrets
key: saslTokenProvider
- parameter: tls
name: keda-kafka-secrets
key: tls
- parameter: awsAccessKeyID
name: keda-kafka-secrets
key: awsAccessKeyID
- parameter: awsSecretAccessKey
name: keda-kafka-secrets
key: awsSecretAccessKey
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: kafka-scaledobject
namespace: default
spec:
scaleTargetRef:
name: azure-functions-deployment
pollingInterval: 30
triggers:
- type: apache-kafka
metadata:
bootstrapServers: localhost:9092
consumerGroup: my-group # Make sure that this consumer group name is the same one as the one that is consuming topics
topic: test-topic
awsRegion: us-east-1 # AWS region of your MSK cluster
# Optional
lagThreshold: "50"
offsetResetPolicy: latest
authenticationRef:
name: keda-trigger-auth-kafka-credential
```

0 comments on commit 7089776

Please sign in to comment.