-
Notifications
You must be signed in to change notification settings - Fork 42
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
#313 solved serializers casting problems using SpecificDatumWriter #315
Merged
davidgarciago
merged 14 commits into
master
from
313-issue-long-timestamp-millis-trouble-to-cast
Dec 19, 2022
Merged
Changes from 9 commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
00c6a76
303 docker compose for securing kafka first attempt
davidgarciago ff75289
303 changed docker compose SSL to PLAINTEXT
davidgarciago de398ad
#303 kafka Auth Configuration
aef053c
#303 added user for producer to connect to kafka
davidgarciago a10d832
#313 solved serializers casting problems using SpecificDatumWritter
davidgarciago ea09ff2
#313 removed unused asignment to DatumWriter
davidgarciago 7a92079
#313 updated pom.xml
davidgarciago 62959b5
Merge branch '303-issue-escaping-with-bar-failing-jaas-auth' of githu…
davidgarciago 2de97e9
#313 datum writer changed to final
davidgarciago 3064978
#313 added new test for serializers with logicaltypes and removed unu…
davidgarciago 9ebb938
#313 Solving checkstyle problems
davidgarciago 9ea2f92
#313 moving avro logical types conversion config to avro serializers
davidgarciago 20ab0fe
#313 checkstyle errors fix
davidgarciago 8ee9b31
#313 changed incorrect pom version
davidgarciago File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -7,7 +7,7 @@ | |
|
||
<artifactId>kloadgen</artifactId> | ||
|
||
<version>5.1.3</version> | ||
<version>5.1.5</version> | ||
|
||
<name>KLoadGen</name> | ||
<description>Load Generation Jmeter plugin for Kafka Cluster. Supporting AVRO, JSON Schema and Protobuf schema types. Generate Artificial data base on Data specification</description> | ||
|
@@ -167,6 +167,17 @@ | |
</roles> | ||
<timezone>Europe/Madrid</timezone> | ||
</developer> | ||
<developer> | ||
<id>davidgarciago</id> | ||
<name>David García Gondell</name> | ||
<email>[email protected]</email> | ||
<organization>Corunet</organization> | ||
<organizationUrl>https://corunet.github.io/</organizationUrl> | ||
<roles> | ||
<role>Junior Developer</role> | ||
</roles> | ||
<timezone>Europe/Madrid</timezone> | ||
</developer> | ||
</developers> | ||
|
||
<scm> | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
KafkaServer { | ||
org.apache.kafka.common.security.plain.PlainLoginModule required | ||
username="admin" | ||
password="admin-secret" | ||
user_admin="admin-secret"; | ||
}; | ||
|
||
Client {}; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
version: '3' | ||
services: | ||
zookeeper: | ||
image: confluentinc/cp-zookeeper | ||
environment: | ||
ZOOKEEPER_CLIENT_PORT: 2181 | ||
ZOOKEEPER_SASL_ENABLED: "false" | ||
kafka: | ||
image: confluentinc/cp-kafka | ||
depends_on: | ||
- zookeeper | ||
ports: | ||
- 29092:29092 | ||
- 9092:9092 | ||
volumes: | ||
- ./conf/kafka/kafka-jaas.conf:/etc/kafka/kafka_server_jaas.conf | ||
environment: | ||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT | ||
KAFKA_LISTENERS: PLAINTEXT://:9092, SASL_PLAINTEXT://:29092 | ||
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,SASL_PLAINTEXT://localhost:29092 | ||
ZOOKEEPER_SASL_ENABLED: "false" | ||
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT | ||
KAFKA_SASL_ENABLED_MECHANISMS: PLAIN | ||
KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN | ||
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 | ||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 | ||
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true" | ||
KAFKA_OPTS: "-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf" | ||
|
||
schema-registry: | ||
image: confluentinc/cp-schema-registry | ||
depends_on: | ||
- zookeeper | ||
- kafka | ||
ports: | ||
- 8081:8081 | ||
volumes: | ||
- ./conf:/conf:ro | ||
environment: | ||
SCHEMA_REGISTRY_HOST_NAME: schema-registry | ||
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092 | ||
SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081 | ||
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181 | ||
SCHEMA_REGISTRY_AUTHENTICATION_METHOD: BASIC | ||
SCHEMA_REGISTRY_AUTHENTICATION_REALM: SchemaRegistry | ||
SCHEMA_REGISTRY_AUTHENTICATION_ROLES: Testers | ||
SCHEMA_REGISTRY_OPTS: -Djava.security.auth.login.config=/conf/schema-registry/schema-registry.jaas | ||
kafka-manager: | ||
image: kafkamanager/kafka-manager | ||
environment: | ||
ZK_HOSTS: zookeeper | ||
ports: | ||
- 9000:9000 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,10 +10,13 @@ | |
import java.io.IOException; | ||
|
||
import lombok.extern.slf4j.Slf4j; | ||
import org.apache.avro.generic.GenericDatumWriter; | ||
import org.apache.avro.generic.GenericRecord; | ||
import org.apache.avro.io.DatumWriter; | ||
import org.apache.avro.io.Encoder; | ||
import org.apache.avro.io.EncoderFactory; | ||
import org.apache.avro.specific.SpecificDatumWriter; | ||
import org.apache.avro.specific.SpecificRecord; | ||
import org.apache.kafka.common.header.Headers; | ||
import org.apache.kafka.common.serialization.Serializer; | ||
|
||
|
@@ -23,7 +26,13 @@ public class GenericAvroRecordSerializer<T extends GenericRecord> implements Ser | |
@Override | ||
public final byte[] serialize(final String topic, final T data) { | ||
|
||
final var writer = new SpecificDatumWriter<>(data.getSchema()); | ||
final DatumWriter<T> writer; | ||
if (data instanceof SpecificRecord) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Missing test |
||
writer = new SpecificDatumWriter<>(data.getSchema()); | ||
} else { | ||
writer = new GenericDatumWriter<>(data.getSchema()); | ||
} | ||
|
||
byte[] dataBytes = new byte[0]; | ||
final var stream = new ByteArrayOutputStream(); | ||
final Encoder jsonEncoder; | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing test