Export Schema Registry client and the serdes functionality (#149)

* Update go modules * Create a SchemaRegistry class to export its functionality to JS * Update tests to reflect changes * Update serdes formats to reflect the changes in its interface * Update serdes functions to reflect the changes to the interface and the way data types and schemas are handled now * Update tests to reflect changes to the serdes functions * Update constants * Rename files to reflect their purpose and have a uniform naming convention * Update writer and reader to remove internal serdes functions * Update tests to reflect the new changes * Clean up tests for topics * Collect all utility functions in utils.go * Add a few helper functions * Update errors and error codes * Update all scripts with the new changes to the serdes and SchemaRegistry * Remove old files, which are either renamed or no longer necessary * Update error codes to remove unnecessary ones * Fix linting issues by removing unnecessary code or fixing reported issues * Rename struct to make it more uniform * Move JS API docs to api-docs directory and update it with latest changes * Update the links to JS API docs in the README * Add test for string serde * Add test for byte array serde * Add docstring comments * Add test for JSON serde * Add test for Avro serde * Add docstring comments * Add docstring comments * Add tests for the JS Connection class * Add tests for the JS SchemaRegistry class * Update path of scripts to the new ones in README * Add tests for the JS Writer class * Add tests for the JS Reader class * Fix topic name * Fix metrics' expected values * Add docstring comment * Update dependencies * Update README with the latest changes
mostafa · Aug 10, 2022 · 90a718f · 90a718f
1 parent 91cf77c
commit 90a718f
Show file tree

Hide file tree

Showing 91 changed files with 3,187 additions and 2,984 deletions.
diff --git a/README.md b/README.md
@@ -14,16 +14,16 @@ If you want to learn more about the extension, see the [article](https://k6.io/b
 
 ## Supported Features
 
-- Produce/consume messages as [String](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_json.js), [stringified JSON](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_json.js), [ByteArray](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_bytes.js), [Avro](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_avro_with_schema_registry.js) and [JSON Schema](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_jsonschema_with_schema_registry.js) format
+- Produce/consume messages as [String](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_json.js), [JSON](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_json.js), [ByteArray](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_bytes.js), [Avro](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_avro_with_schema_registry.js) and [JSON Schema](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_jsonschema_with_schema_registry.js) formats
 - Support for user-provided [Avro](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_avro.js) and [JSON Schema](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_jsonschema_with_schema_registry.js) key and value schemas in the script
-- Authentication with [SASL PLAIN and SCRAM](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_sasl_auth.js)
+- Authentication with [SASL PLAIN, SCRAM and SSL](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_sasl_auth.js)
 - Create, list and delete [topics](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_topics.js)
 - Support for loading Avro schemas from [Schema Registry](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_avro_with_schema_registry.js)
 - Support for [byte array](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_bytes.js) for binary data (from binary protocols)
 - Support consumption from all partitions with a group ID
-- Support Kafka message compression: Gzip, [Snappy](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_json_with_snappy_compression.js), Lz4 & Zstd
-- Support for sending messages with [no key](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_avro_no_key.js)
-- Support for k6 [thresholds](https://github.com/mostafa/xk6-kafka/blob/e1a810d52112f05d7a66c12740d9885ebb64897e/scripts/test_json.js#L21-L27) on custom Kafka metrics
+- Support Kafka message compression: Gzip, [Snappy](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_json.js), Lz4 & Zstd
+- Support for sending messages with [no key](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_avro_no_schema_registry.js)
+- Support for k6 [thresholds](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_json.js) on custom Kafka metrics
 - Support for [headers](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_json.js) on produced and consumed messages
 - Lots of exported metrics, as shown in the result output of the [k6 test script](https://github.com/mostafa/xk6-kafka/blob/main/README.md#k6-test-script)
 
@@ -127,7 +127,7 @@ I recommend the [fast-data-dev](https://github.com/lensesio/fast-data-dev) Docke
 
 ### The xk6-kafka API
 
-All the exported functions are available by importing the module object from `k6/x/kafka`. The exported objects, constants and other data structures are available in the [`index.d.ts`](https://github.com/mostafa/xk6-kafka/blob/main/index.d.ts) file, and they always reflect the *latest* changes on the `main` branch. You can access the generated documentation at [`docs/README.md`](https://github.com/mostafa/xk6-kafka/blob/main/docs/README.md).
+All the exported functions are available by importing the module object from `k6/x/kafka`. The exported objects, constants and other data structures are available in the [`index.d.ts`](https://github.com/mostafa/xk6-kafka/blob/main/index.d.ts) file, and they always reflect the *latest* changes on the `main` branch. You can access the generated documentation at [`api-docs/docs/README.md`](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/docs/README.md).
 
 > ⚠️ **Warning:**
 > The Javascript API is subject to change in future versions unless a new major version is released.
@@ -143,7 +143,7 @@ The example scripts are available as `test_<format/feature>.js` with more code a
     import * as kafka from "k6/x/kafka";
 
     // Or individual classes and constants
-    import { Writer, Reader, Connection, SOME_CONSTANT } from "k6/x/kafka";
+    import { Writer, Reader, Connection, SchemaRegistry, SCHEMA_TYPE_STRING } from "k6/x/kafka";
     ```
 
 2. You need to instantiate the classes in the `init` context. All the [k6 options](https://k6.io/docs/using-k6/k6-options/) are also configured here:
@@ -167,6 +167,10 @@ The example scripts are available as `test_<format/feature>.js` with more code a
         address: "localhost:9092",
     });
 
+    const schemaRegistry = new SchemaRegistry(
+        // Can accept a SchemaRegistryConfig object
+    )
+
     if (__VU == 0) {
         // Create a topic on initialization (before producing messages)
         connection.createTopic({
@@ -190,8 +194,14 @@ The example scripts are available as `test_<format/feature>.js` with more code a
             messages: [
                 // Message object(s)
                 {
-                    key: "my-key",
-                    value: "my-value",
+                    key: schemaRegistry.serialize({
+                        data: "my-key",
+                        schemaType: SCHEMA_TYPE_STRING,
+                    }),
+                    value: schemaRegistry.serialize({
+                        data: "my-value",
+                        schemaType: SCHEMA_TYPE_STRING,
+                    }),
                 },
             ],
         });
@@ -203,10 +213,18 @@ The example scripts are available as `test_<format/feature>.js` with more code a
         });
 
         // your messages
-        console.log(message);
+        console.log(messages);
 
         // You can use checks to verify the contents,
         // length and other properties of the message(s)
+
+        // To serialize the data back into a string, you should use
+        // the deserialize method of the Schema Registry client. You
+        // can use it inside a check, as shown in the example scripts.
+        let deserializedValue = schemaRegistry.deserialize({
+            data: messages[0].value,
+            schemaType: SCHEMA_TYPE_STRING,
+        })
     }
     ```
 
@@ -234,11 +252,11 @@ The example scripts are available as `test_<format/feature>.js` with more code a
 
     ```bash
 
-              /\      |‾‾| /‾‾/   /‾‾/
-         /\  /  \     |  |/  /   /  /
-        /  \/    \    |     (   /   ‾‾\
-       /          \   |  |\  \ |  (‾)  |
-      / __________ \  |__| \__\ \_____/ .io
+            /\      |‾‾| /‾‾/   /‾‾/
+       /\  /  \     |  |/  /   /  /
+      /  \/    \    |     (   /   ‾‾\
+     /          \   |  |\  \ |  (‾)  |
+    / __________ \  |__| \__\ \_____/ .io
 
     execution: local
         script: scripts/test_json.js
@@ -248,13 +266,13 @@ The example scripts are available as `test_<format/feature>.js` with more code a
             * default: 50 looping VUs for 1m0s (gracefulStop: 30s)
 
 
-    running (1m00.2s), 00/50 VUs, 13778 complete and 0 interrupted iterations
+    running (1m04.4s), 00/50 VUs, 20170 complete and 0 interrupted iterations
     default ✓ [======================================] 50 VUs  1m0s
 
         ✓ 10 messages are received
         ✓ Topic equals to xk6_kafka_json_topic
-        ✓ Key is correct
-        ✓ Value is correct
+        ✓ Key contains key/value and is JSON
+        ✓ Value contains key/value and is JSON
         ✓ Header equals {'mykey': 'myvalue'}
         ✓ Time is past
         ✓ Partition is zero
@@ -263,47 +281,47 @@ The example scripts are available as `test_<format/feature>.js` with more code a
 
         █ teardown
 
-        checks.........................: 100.00% ✓ 124002       ✗ 0
+        checks.........................: 100.00% ✓ 181530       ✗ 0
         data_received..................: 0 B     0 B/s
         data_sent......................: 0 B     0 B/s
-        iteration_duration.............: avg=217.98ms min=26.64ms med=216.88ms max=357.64ms p(90)=244.95ms p(95)=254.86ms
-        iterations.....................: 13778   229.051752/s
-        kafka.reader.dial.count........: 50      0.831223/s
-        kafka.reader.dial.seconds......: avg=4.76µs   min=0s      med=0s       max=2.22ms   p(90)=0s       p(95)=0s
+        iteration_duration.............: avg=153.45ms min=6.01ms med=26.8ms  max=8.14s   p(90)=156.3ms p(95)=206.4ms
+        iterations.....................: 20170   313.068545/s
+        kafka.reader.dial.count........: 50      0.776075/s
+        kafka.reader.dial.seconds......: avg=171.22µs min=0s     med=0s      max=1.09s   p(90)=0s      p(95)=0s
      ✓ kafka.reader.error.count.......: 0       0/s
         kafka.reader.fetch_bytes.max...: 1000000 min=1000000    max=1000000
         kafka.reader.fetch_bytes.min...: 1       min=1          max=1
         kafka.reader.fetch_wait.max....: 200ms   min=200ms      max=200ms
-        kafka.reader.fetch.bytes.......: 0 B     0 B/s
-        kafka.reader.fetch.size........: 0       0/s
-        kafka.reader.fetches.count.....: 50      0.831223/s
-        kafka.reader.lag...............: 7457    min=5736       max=14370
-        kafka.reader.message.bytes.....: 27 MB   450 kB/s
-        kafka.reader.message.count.....: 137830  2291.348744/s
-        kafka.reader.offset............: 2740    min=11         max=2810
+        kafka.reader.fetch.bytes.......: 58 MB   897 kB/s
+        kafka.reader.fetch.size........: 147167  2284.25179/s
+        kafka.reader.fetches.count.....: 107     1.6608/s
+        kafka.reader.lag...............: 1519055 min=0          max=2436190
+        kafka.reader.message.bytes.....: 40 MB   615 kB/s
+        kafka.reader.message.count.....: 201749  3131.446006/s
+        kafka.reader.offset............: 4130    min=11         max=5130
         kafka.reader.queue.capacity....: 1       min=1          max=1
         kafka.reader.queue.length......: 1       min=0          max=1
-        kafka.reader.read.seconds......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s
+        kafka.reader.read.seconds......: avg=96.5ms   min=0s     med=0s      max=59.37s  p(90)=0s      p(95)=0s
         kafka.reader.rebalance.count...: 0       0/s
-        kafka.reader.timeouts.count....: 0       0/s
-        kafka.reader.wait.seconds......: avg=7.44µs   min=0s      med=0s       max=3.17ms   p(90)=0s       p(95)=0s
-        kafka.writer.acks.required.....: -1      min=-1         max=0
-        kafka.writer.async.............: 0.00%   ✓ 0            ✗ 1377800
+        kafka.reader.timeouts.count....: 57      0.884725/s
+        kafka.reader.wait.seconds......: avg=102.71µs min=0s     med=0s      max=85.71ms p(90)=0s      p(95)=0s
+        kafka.writer.acks.required.....: 0       min=0          max=0
+        kafka.writer.async.............: 0.00%   ✓ 0            ✗ 2017000
         kafka.writer.attempts.max......: 0       min=0          max=0
-        kafka.writer.batch.bytes.......: 302 MB  5.0 MB/s
+        kafka.writer.batch.bytes.......: 441 MB  6.8 MB/s
         kafka.writer.batch.max.........: 1       min=1          max=1
-        kafka.writer.batch.size........: 1377800 22905.17521/s
+        kafka.writer.batch.size........: 2017000 31306.854525/s
         kafka.writer.batch.timeout.....: 0s      min=0s         max=0s
-     ✓ kafka.writer.error.count.......: 0       0/s
-        kafka.writer.message.bytes.....: 603 MB  10 MB/s
-        kafka.writer.message.count.....: 2755600 45810.350421/s
+      ✓ kafka.writer.error.count.......: 0       0/s
+        kafka.writer.message.bytes.....: 883 MB  14 MB/s
+        kafka.writer.message.count.....: 4034000 62613.709051/s
         kafka.writer.read.timeout......: 0s      min=0s         max=0s
         kafka.writer.retries.count.....: 0       0/s
-        kafka.writer.wait.seconds......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s
-        kafka.writer.write.count.......: 2755600 45810.350421/s
-        kafka.writer.write.seconds.....: avg=1.02ms   min=79.29µs med=893.09µs max=24.26ms  p(90)=1.22ms   p(95)=1.74ms
+        kafka.writer.wait.seconds......: avg=0s       min=0s     med=0s      max=0s      p(90)=0s      p(95)=0s
+        kafka.writer.write.count.......: 4034000 62613.709051/s
+        kafka.writer.write.seconds.....: avg=523.21µs min=4.84µs med=14.48µs max=4.05s   p(90)=33.85µs p(95)=42.68µs
         kafka.writer.write.timeout.....: 0s      min=0s         max=0s
-        vus............................: 50      min=50         max=50
+        vus............................: 7       min=7          max=50
         vus_max........................: 50      min=50         max=50
     ```
 
@@ -326,6 +344,8 @@ $ docker exec -it lensesio bash
 > **Note:**
 > If you want to test SASL authentication, look at [this commit message](https://github.com/mostafa/xk6-kafka/pull/3/commits/403fbc48d13683d836b8033eeeefa48bf2f25c6e), where I describe how to run a test environment.
 
+Also, if the `reader.consume` keeps hanging, it might be because the topic doesn't exist or is empty.
+
 ## Contributions, Issues and Feedback
 
 I'd be thrilled to receive contributions and feedback on this project. You're always welcome to create an issue if you find one (or many). I would do my best to address the issues. Also, feel free to contribute by opening a PR with changes, and I'll do my best to review and merge it as soon as I can.

diff --git a/docs/.nojekyll → api-docs/.nojekyll b/docs/.nojekyll → api-docs/.nojekyll
diff --git a/api-docs/docs/.nojekyll b/api-docs/docs/.nojekyll
@@ -0,0 +1 @@
+TypeDoc added this file to prevent GitHub Pages from using Jekyll. You can turn off this behavior by setting the `githubPages` option to false.
diff --git a/docs/README.md → api-docs/docs/README.md b/docs/README.md → api-docs/docs/README.md
@@ -12,35 +12,36 @@ This documentation refers to the development version of the xk6-kafka project, w
 
 - [BALANCERS](enums/BALANCERS.md)
 - [COMPRESSION\_CODECS](enums/COMPRESSION_CODECS.md)
-- [DESERIALIZERS](enums/DESERIALIZERS.md)
+- [ELEMENT\_TYPES](enums/ELEMENT_TYPES.md)
 - [GROUP\_BALANCERS](enums/GROUP_BALANCERS.md)
 - [ISOLATION\_LEVEL](enums/ISOLATION_LEVEL.md)
 - [SASL\_MECHANISMS](enums/SASL_MECHANISMS.md)
-- [SERIALIZERS](enums/SERIALIZERS.md)
+- [SCHEMA\_TYPES](enums/SCHEMA_TYPES.md)
 - [SUBJECT\_NAME\_STRATEGY](enums/SUBJECT_NAME_STRATEGY.md)
 - [TLS\_VERSIONS](enums/TLS_VERSIONS.md)
 
 ### Classes
 
 - [Connection](classes/Connection.md)
 - [Reader](classes/Reader.md)
+- [SchemaRegistry](classes/SchemaRegistry.md)
 - [Writer](classes/Writer.md)
 
 ### Interfaces
 
 - [BasicAuth](interfaces/BasicAuth.md)
 - [ConfigEntry](interfaces/ConfigEntry.md)
-- [Configuration](interfaces/Configuration.md)
 - [ConnectionConfig](interfaces/ConnectionConfig.md)
 - [ConsumeConfig](interfaces/ConsumeConfig.md)
-- [ConsumerConfiguration](interfaces/ConsumerConfiguration.md)
+- [Container](interfaces/Container.md)
 - [Message](interfaces/Message.md)
 - [ProduceConfig](interfaces/ProduceConfig.md)
-- [ProducerConfiguration](interfaces/ProducerConfiguration.md)
 - [ReaderConfig](interfaces/ReaderConfig.md)
 - [ReplicaAssignment](interfaces/ReplicaAssignment.md)
 - [SASLConfig](interfaces/SASLConfig.md)
-- [SchemaRegistryConfiguration](interfaces/SchemaRegistryConfiguration.md)
+- [Schema](interfaces/Schema.md)
+- [SchemaRegistryConfig](interfaces/SchemaRegistryConfig.md)
+- [SubjectNameConfig](interfaces/SubjectNameConfig.md)
 - [TLSConfig](interfaces/TLSConfig.md)
 - [TopicConfig](interfaces/TopicConfig.md)
 - [WriterConfig](interfaces/WriterConfig.md)
diff --git a/docs/classes/Connection.md → api-docs/docs/classes/Connection.md b/docs/classes/Connection.md → api-docs/docs/classes/Connection.md
@@ -1,6 +1,6 @@
 # Class: Connection
 
-**`classdesc`** Connection can connect to Kafka for working with topics.
+**`classdesc`** Connection connects to Kafka for working with topics.
 
 **`example`**
 
@@ -44,7 +44,7 @@ connection.close();
 
 #### Defined in
 
-[index.d.ts:365](https://github.com/mostafa/xk6-kafka/blob/main/index.d.ts#L365)
+index.d.ts:353
 
 ## Methods
 
@@ -64,7 +64,7 @@ connection.close();
 
 #### Defined in
 
-[index.d.ts:391](https://github.com/mostafa/xk6-kafka/blob/main/index.d.ts#L391)
+index.d.ts:379
 
 ___
 
@@ -89,7 +89,7 @@ Create a new topic.
 
 #### Defined in
 
-[index.d.ts:372](https://github.com/mostafa/xk6-kafka/blob/main/index.d.ts#L372)
+index.d.ts:360
 
 ___
 
@@ -114,7 +114,7 @@ Delete a topic.
 
 #### Defined in
 
-[index.d.ts:379](https://github.com/mostafa/xk6-kafka/blob/main/index.d.ts#L379)
+index.d.ts:367
 
 ___
 
@@ -133,4 +133,4 @@ List topics.
 
 #### Defined in
 
-[index.d.ts:385](https://github.com/mostafa/xk6-kafka/blob/main/index.d.ts#L385)
+index.d.ts:373
diff --git a/docs/classes/Reader.md → api-docs/docs/classes/Reader.md b/docs/classes/Reader.md → api-docs/docs/classes/Reader.md
@@ -1,6 +1,6 @@
 # Class: Reader
 
-**`classdesc`** Reader can read messages from Kafka.
+**`classdesc`** Reader reads messages from Kafka.
 
 **`example`**
 
@@ -43,7 +43,7 @@ reader.close();
 
 #### Defined in
 
-[index.d.ts:324](https://github.com/mostafa/xk6-kafka/blob/main/index.d.ts#L324)
+index.d.ts:312
 
 ## Methods
 
@@ -63,7 +63,7 @@ reader.close();
 
 #### Defined in
 
-[index.d.ts:337](https://github.com/mostafa/xk6-kafka/blob/main/index.d.ts#L337)
+index.d.ts:325
 
 ___
 
@@ -88,4 +88,4 @@ Read messages from Kafka.
 
 #### Defined in
 
-[index.d.ts:331](https://github.com/mostafa/xk6-kafka/blob/main/index.d.ts#L331)
+index.d.ts:319
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		TypeDoc added this file to prevent GitHub Pages from using Jekyll. You can turn off this behavior by setting the `githubPages` option to false.