getting-started-kafka

Getting started with OpenShift Streams for Apache Kafka

As a developer of applications and services, you can use OpenShift Streams for Apache Kafka to create and set up Kafka instances and connect your applications and services to these instances. Streams for Apache Kafka is a managed cloud service that enables you to add Kafka data-streaming functionality in your applications without having to install, configure, run, and maintain your own Kafka clusters.

Creating a Kafka instance in Streams for Apache Kafka

Use the OpenShift Streams for Apache Kafka web console to create and configure a Kafka instance for your applications or services. A Kafka instance in Streams for Apache Kafka includes a Kafka cluster, bootstrap server, and the configuration details needed for producers and consumers to connect to the instance.

Prerequisites

• You’re logged in to the Streams for Apache Kafka web console at https://console.redhat.com/application-services/streams/.

• You’re familiar with the different Kafka instance sizes that you can create in Streams for Apache Kafka, and the capacities that correspond to those sizes. For more information about the capacities and service limits of your Kafka instance, see Red Hat OpenShift Streams for Apache Kafka Service Limits.

Procedure

1. On the Kafka Instances page of the Streams for Apache Kafka web console, click Create Kafka instance.

2. Complete the Create a Kafka instance form to define the following instance details. Some instance details currently have only one option.

   • Instance name: Enter a unique name for the Kafka instance, such as my-first-kafka-instance.

   • Cloud provider: Accept the default value of Amazon Web Services.

   • Cloud region: Select a cloud provider region from the list.

   • Availability zones: The selection for availability zones is preconfigured as Multi for a production instance and Single for a trial instance.

   • Size: If you’ve purchased a subscription for Streams for Apache Kafka, use the slider to select an instance size. In Streams for Apache Kafka, you can create Kafka instances of 1 or 2 streaming units depending on the capacity required. If you’re creating a trial instance, the instance size is preconfigured.

3. Click Create instance to start the creation process for your Kafka instance.

   The new Kafka instance is listed on the Kafka Instances page. Typically, you need to wait a few minutes for the instance creation process to finish.

   When the instance has a status of Ready, you can start using the instance. As shown in the following figure, you can click the options icon (three vertical dots) to view instance and connection details, change the instance owner, or delete the instance.

   

   Figure 1. Kafka instance options menu

   Note

   Although you can see Kafka instances created by other users in your organization, you might not be able to manage or connect to those instances. Only the instance owner or users with permissions to access the instance can edit or delete the instance, access the associated service account and topics, or connect to the instance.

Verification

1. Verify that the new Kafka instance is listed on the Kafka Instances page.

2. Verify that the status of the new Kafka instance is shown as Ready.

Creating a service account to connect to a Kafka instance in OpenShift Streams for Apache Kafka

To connect your applications or services to a Kafka instance in OpenShift Streams for Apache Kafka, you need to create a service account that’s associated with the instance. You then need to save the generated service account credentials, the authentication token endpoint, and the bootstrap server endpoint for the instance to a secure location. You’ll use these details when you configure an application to connect to the Kafka instance.

Prerequisites

• You have a running Kafka instance in Streams for Apache Kafka.

Procedure

1. On the Kafka Instances page of the Streams for Apache Kafka web console, select the options icon (three vertical dots) for your Kafka instance and click Connection.

2. On the Connection page, copy the Bootstrap server endpoint to a secure location. You’ll specify this endpoint when configuring a connection to the Kafka instance.

   The remainder of this section describes how to create a service account and copy the generated credentials. If you want to use the credentials of an existing service account, you can skip to the next section.

3. Open the Service Accounts page in the Application Services section of the Red Hat Hybrid Cloud Console.

4. Click Create service account to set up the account that you’ll use to access this Kafka instance.

5. Enter a short description such as my-service-account and click Create.

6. Copy the generated Client ID and Client secret values to a secure location. You’ll specify these credentials when configuring a connection to the Kafka instance.

   Important

   The generated credentials are displayed only once, so ensure that you’ve successfully and securely saved the copied credentials before closing the credentials window.

7. After you save the generated credentials to a secure location, select the confirmation check box in the credentials window and close the window.

8. Under Authentication method, copy the SASL/OAUTHBEARER Token endpoint URL value to a secure location. This is the endpoint that you’ll use with your service account credentials to authenticate the connection to the Kafka instance.

   Note	SASL/PLAIN authentication is also available for tools and libraries that don’t support SASL/OAUTHBEARER, but SASL/OAUTHBEARER is recommended whenever possible. With SASL/PLAIN authentication, you use only service account credentials to authenticate the connection to the Kafka instance.

9. (Optional) To review, reset, or delete the service account, use the Service Accounts page.

Verification

1. Verify that the bootstrap server, client credentials, and authentication token endpoint are saved to a secure location.

2. Verify that your service account was successfully created on the Service Accounts page.

Setting permissions for a service account in a Kafka instance in Streams for Apache Kafka

After you create a service account to connect to a Kafka instance, you must also set the appropriate level of access for that new account in an Access Control List (ACL) for the Kafka instance. OpenShift Streams for Apache Kafka uses ACLs provided by Kafka that enable you to manage how other user accounts and service accounts are permitted to interact with the Kafka resources that you create.

Prerequisites

• You have a running Kafka instance in Streams for Apache Kafka.

• You’ve created a service account that you want to allow to access the running Kafka instance.

Procedure

1. In the OpenShift Streams for Apache Kafka web console, select Kafka Instances and then click the name of the Kafka instance that you want the service account to access.

2. Click the Access tab to view the current ACL for this instance.

3. To modify the ACL, click Manage access.

4. In the Manage access dialog box, use the Account list to select the service account that you previously created, and click Next.

5. Under Assign Permissions, use the list to select the Consume from a topic and the Produce to a topic permission options, and set all resource identifiers to Is and all identifier values to *.

   These settings result in the following ACL permissions for the new service account:

   Table 1. Example ACL permissions for a new service account

   Resource type	Resource identifier and value	Access type	Operation
   Topic (For consuming)	Is = *	Allow	Read, Describe
   Consumer group (For consuming)	Is = *	Allow	Read
   Topic (For producing)	Is = *	Allow	Write, Create, Describe

   The permissions shown in the table enable applications associated with the service account to create topics in the Kafka instance, to produce and consume messages in any topic in the instance, and to use any consumer group.

   Note

   Alternatively, you can click Add permission to create individual permissions as needed. For example, you can create one Topic entry and one Consumer group entry, both with Allow access to All operations. This enables both consuming and producing for the specified topic in a single entry, and enables all permissions for the consumer group in another single entry. But you must configure these entries individually without using the predefined permission options.

6. After you add these permissions for the service account, click Save to finish.

Verification

• Verify that the new permissions for the service account are listed on the Access page of the Kafka instance.

Additional resources

• Managing account access in OpenShift Streams for Apache Kafka

• Authorization and ACLs in the Apache Kafka documentation

Creating a Kafka topic in Streams for Apache Kafka

After you create a Kafka instance, you can create Kafka topics to start producing and consuming messages in your applications and services.

Prerequisites

• You have a running Kafka instance in OpenShift Streams for Apache Kafka.

Procedure

1. In the OpenShift Streams for Apache Kafka web console, select Kafka Instances and then click the name of the Kafka instance that you want to add a topic to.

2. Select the Topics tab.

3. Click Create topic and follow the guided steps to define the topic details, as shown in the figure.

   

   Figure 2. Guided steps to define topic details

   You must specify the following topic properties:

   • Topic name: Enter a unique topic name, such as my-first-kafka-topic.

   • Partitions: Set the number of partitions for this topic. This example sets the partitions value to 1. Partitions are distinct lists of messages within a topic and enable parts of a topic to be distributed over multiple brokers in the cluster. A topic can contain one or more partitions, enabling producer and consumer loads to be scaled.

   • Message retention: Set the message retention time and size to the relevant value and increment. This example sets the retention time to A week and the retention size to Unlimited. Message retention time is the amount of time that messages are retained in a topic before they are deleted or compacted, depending on the cleanup policy. Retention size is the maximum total size of all log segments in a partition before they are deleted or compacted.

   • Replicas: Replicas are copies of partitions in a topic. For this release of Streams for Apache Kafka, the replica values are preconfigured. For a standard Kafka instance, the number of partition replicas for the topic is set to 3 and the minimum number of follower replicas that must be in sync with a partition leader is set to 2. For a trial Kafka instance, the number of replicas and the minimum in-sync replica factor are both set to 1. Partition replicas are distributed over multiple brokers in the cluster to ensure topic availability if a broker fails. When a follower replica is in sync with a partition leader, the follower replica can become the new partition leader if needed.

   After you complete the topic setup, the new topic is listed on the Topics page. You can now start producing and consuming messages to and from this topic using applications that you connect to the Kafka instance.

   Note	If the topic creation is unsuccessful and you see a 400 Bad Request error message, try to create your topic again later. This situation might occur, for example, if your selected cloud provider has a temporary availability problem that affects your Kafka instance.

4. (Optional) To edit or delete the topic, click the options icon (three vertical dots) next to the topic name, as shown in the figure.

Verification

• Verify that the new Kafka topic is listed on the Topics page.

Additional resources

• Purchase a subscription to OpenShift Streams for Apache Kafka

• Configuring topics in OpenShift Streams for Apache Kafka

• Getting started with the rhoas CLI for OpenShift Streams for Apache Kafka

• CLI command reference (rhoas)

• Configuring and connecting Kcat with OpenShift Streams for Apache Kafka

• Configuring and connecting Kafka scripts with OpenShift Streams for Apache Kafka

• Using Quarkus applications with Kafka instances in OpenShift Streams for Apache Kafka