kafka sandbox example #9762

amre-abb · 2020-01-21T18:40:56Z

Description: Add a simple sandbox example for kafka filter
Risk Level: Low
Testing: Built docs locally and verified the generated docs.
Docs Changes: Yes
Release Notes: Add a sandbox example for kafka filter.
[Optional Fixes #Issue]
[Optional Deprecated:]

lucperkins · 2020-01-21T19:37:03Z

@amre-abb Please provide a Developer Certificate of Origin

mattklein123 · 2020-01-21T23:40:26Z

This is awesome! @adamkotwasinski PTAL

adamkotwasinski · 2020-01-21T23:55:20Z

LGTM (thank you!)

amre-abb · 2020-01-21T23:58:57Z

Thanks @adamkotwasinski and @mattklein123 for a quick review. I will look at the CI/build failures and address @lucperkins comment soon.

P.S. @adamkotwasinski thanks a lot for writing a kafka filter! it is awesome!

mattklein123

Thank you so much for this! One small comment.

/wait

examples/kafka/envoy.yaml

moderation · 2020-01-22T18:22:46Z

examples/kafka/envoy.yaml

+    - filters:
+      - name: envoy.filters.network.kafka_broker
+        typed_config:
+          "@type": type.googleapis.com/envoy.config.filter.network.kafka_broker.v2alpha1.KafkaBroker


I know this is jumping a bit outside of this PR for the sandbox but the actual Kafka documentation does not use the proper typed_config which it should. RST to be updated at https://github.com/envoyproxy/envoy/blob/master/docs/root/configuration/listeners/network_filters/kafka_broker_filter.rst. I can open a separate PR or ask @adamkotwasinski to add this to his Kafka version update PR at #9582

👍 Thanks! I'll fix it there

Signed-off-by: amre_shakimov <[email protected]>

... and move kafka.rst file to a correct location Signed-off-by: amre_shakimov <[email protected]>

Signed-off-by: amre_shakimov <[email protected]>

mattklein123 · 2020-01-23T21:44:35Z

@amre-abb friendly request to not force push, it breaks review history. Just merge master and add commits. Thanks!

mattklein123 · 2020-01-24T20:41:05Z

Very close now but you have a deprecated field to fix: https://dev.azure.com/cncf/envoy/_build/results?buildId=28392&view=logs&j=ec9f720e-f904-5f65-4ba4-f371a013344f&t=ee47c280-92b1-5c80-3a10-d3d23a58a062&l=2294

Thank you!

/wait

Signed-off-by: amre_shakimov <[email protected]>

moderation · 2020-01-25T00:22:59Z

docs/root/start/sandboxes/kafka.rst

+
+.. code-block:: console
+
+  $ docker run --rm -it --network envoymesh wurstmeister/kafka:2.11-2.0.0 /opt/kafka/bin/kafka-console-producer.sh --broker-list envoy:19092 --topic test_topic


The Kafka filter currently supports Kafka version 2.3.1 in master - #9582). The Docker image versions you have here is 2.11 Scala / 2.0.0 Kafka which are about a year old. Should we setup the sandbox using the latest Kafka 2.3.1 container at wurstmeister/kafka:2.12-2.3.1 ?

2.3.1 would be a safer bet, 2.4.0 is going to involve more C++ work than usual
(later on I can just grep for 2.3.1 in whole repo and make PRs for review by other folks so we don't get out of sync)

Agree. I just tested the sandbox with 2.3.1 and it works great. Have updated my comment.

Thanks, I have the change locally. Will push after this CI run finishes just to make sure there are no issues.

moderation · 2020-01-25T00:23:20Z

docs/root/start/sandboxes/kafka.rst

+
+.. code-block:: console
+
+  $ docker run --rm -it --network envoymesh wurstmeister/kafka:2.11-2.0.0 /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server envoy:19092 --topic test_topic --from-beginning


See comment above re: version

moderation · 2020-01-25T00:24:12Z

examples/kafka/docker-compose.yaml

+    - "2181"
+
+  kafka:
+    image: wurstmeister/kafka:2.11-2.0.0


See comment above re: version

moderation · 2020-01-25T00:59:32Z

examples/kafka/docker-compose.yaml

+      - "8001:8001"
+
+  zookeeper:
+    image: wurstmeister/zookeeper:3.4.6


Update to zookeeper:3.5.6 which is is the official certified Zookeeper image and has a nice side effect of being more than half the size of the old wurstmeister image. https://hub.docker.com/_/zookeeper?tab=tags. I've confirmed this image works in the sandbox in testing.

I am reluctant to use latest tags in general. And the version difference is not very significant so I'd prefer to leave it as is unless you have a strong opinion about this.

I just updated my comment to suggest using the official certified Zookeeper image. Agree with avoiding :latest. I tested with this image and 👍

mattklein123 · 2020-01-26T18:08:22Z

@amre-abb sorry looks like DCO is broken. Can you do a force push to fix and we can get this merged? Thank you!

/wait

Signed-off-by: amre_shakimov <[email protected]>

mattklein123 · 2020-01-27T17:06:05Z

DCO still needs fixing. :(

/wait

stale · 2020-02-03T17:58:26Z

This pull request has been automatically marked as stale because it has not had activity in the last 7 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

stale · 2020-02-10T18:28:27Z

This pull request has been automatically closed because it has not had activity in the last 14 days. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

amre-abb force-pushed the amre--add-kafka-sandbox-example branch 3 times, most recently from 88c39f8 to 1a5a448 Compare January 21, 2020 19:26

mattklein123 assigned adamkotwasinski Jan 21, 2020

amre-abb force-pushed the amre--add-kafka-sandbox-example branch 2 times, most recently from d621233 to c6dc4b8 Compare January 22, 2020 16:30

mattklein123 self-assigned this Jan 22, 2020

mattklein123 requested changes Jan 22, 2020

View reviewed changes

examples/kafka/envoy.yaml Show resolved Hide resolved

repokitteh-read-only bot added waiting and removed waiting labels Jan 22, 2020

amre-abb force-pushed the amre--add-kafka-sandbox-example branch from e4db965 to 74f9c15 Compare January 22, 2020 17:37

moderation reviewed Jan 22, 2020

View reviewed changes

mattklein123 added the waiting label Jan 22, 2020

amre-abb added 5 commits January 23, 2020 10:35

initial working kafka example

dbf415e

Signed-off-by: amre_shakimov <[email protected]>

fix topic name is consuming command

c954945

... and move kafka.rst file to a correct location Signed-off-by: amre_shakimov <[email protected]>

move kafka under examples

bcb92df

Signed-off-by: amre_shakimov <[email protected]>

formatting

228cba3

Signed-off-by: amre_shakimov <[email protected]>

add kafka/envoy.yml to BUILD

32f8ad6

Signed-off-by: amre_shakimov <[email protected]>

amre-abb force-pushed the amre--add-kafka-sandbox-example branch from 74f9c15 to 32f8ad6 Compare January 23, 2020 18:38

repokitteh-read-only bot removed the waiting label Jan 23, 2020

mattklein123 added the waiting label Jan 23, 2020

repokitteh-read-only bot removed the waiting label Jan 23, 2020

fix example_configs_test

e7f19b0

Signed-off-by: amre_shakimov <[email protected]>

amre-abb force-pushed the amre--add-kafka-sandbox-example branch from b8cebf2 to e7f19b0 Compare January 23, 2020 21:38

repokitteh-read-only bot added the waiting label Jan 24, 2020

use typed_config instead of a deprecated option config for filters

ff73238

repokitteh-read-only bot removed the waiting label Jan 24, 2020

amre-abb added 2 commits January 24, 2020 14:48

remove access log from tcp filter

e7949f1

Signed-off-by: amre_shakimov <[email protected]>

do not use a deprecated option hosts

4b50894

Signed-off-by: amre_shakimov <[email protected]>

moderation reviewed Jan 25, 2020

View reviewed changes

repokitteh-read-only bot added the waiting label Jan 26, 2020

use kafka:2.12-2.3.1 and zookeeper:3.5.6 in the sandbox example

5d8b13e

Signed-off-by: amre_shakimov <[email protected]>

amre-abb force-pushed the amre--add-kafka-sandbox-example branch from ba1b901 to 5d8b13e Compare January 26, 2020 18:47

repokitteh-read-only bot removed the waiting label Jan 26, 2020

repokitteh-read-only bot added the waiting label Jan 27, 2020

stale bot added the stale stalebot believes this issue/PR has not been touched recently label Feb 3, 2020

stale bot closed this Feb 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kafka sandbox example #9762

kafka sandbox example #9762

amre-abb commented Jan 21, 2020

lucperkins commented Jan 21, 2020

mattklein123 commented Jan 21, 2020

adamkotwasinski commented Jan 21, 2020

amre-abb commented Jan 21, 2020

mattklein123 left a comment

moderation Jan 22, 2020 •

edited

Loading

adamkotwasinski Jan 22, 2020

mattklein123 commented Jan 23, 2020

mattklein123 commented Jan 24, 2020

moderation Jan 25, 2020 •

edited

Loading

adamkotwasinski Jan 25, 2020 •

edited

Loading

moderation Jan 25, 2020

amre-abb Jan 25, 2020

moderation Jan 25, 2020

moderation Jan 25, 2020

moderation Jan 25, 2020 •

edited

Loading

amre-abb Jan 25, 2020

moderation Jan 25, 2020

mattklein123 commented Jan 26, 2020

mattklein123 commented Jan 27, 2020

stale bot commented Feb 3, 2020

stale bot commented Feb 10, 2020


		.. code-block:: console

		$ docker run --rm -it --network envoymesh wurstmeister/kafka:2.11-2.0.0 /opt/kafka/bin/kafka-console-producer.sh --broker-list envoy:19092 --topic test_topic

kafka sandbox example #9762

kafka sandbox example #9762

Conversation

amre-abb commented Jan 21, 2020

lucperkins commented Jan 21, 2020

mattklein123 commented Jan 21, 2020

adamkotwasinski commented Jan 21, 2020

amre-abb commented Jan 21, 2020

mattklein123 left a comment

Choose a reason for hiding this comment

moderation Jan 22, 2020 • edited Loading

Choose a reason for hiding this comment

adamkotwasinski Jan 22, 2020

Choose a reason for hiding this comment

mattklein123 commented Jan 23, 2020

mattklein123 commented Jan 24, 2020

moderation Jan 25, 2020 • edited Loading

Choose a reason for hiding this comment

adamkotwasinski Jan 25, 2020 • edited Loading

Choose a reason for hiding this comment

moderation Jan 25, 2020

Choose a reason for hiding this comment

amre-abb Jan 25, 2020

Choose a reason for hiding this comment

moderation Jan 25, 2020

Choose a reason for hiding this comment

moderation Jan 25, 2020

Choose a reason for hiding this comment

moderation Jan 25, 2020 • edited Loading

Choose a reason for hiding this comment

amre-abb Jan 25, 2020

Choose a reason for hiding this comment

moderation Jan 25, 2020

Choose a reason for hiding this comment

mattklein123 commented Jan 26, 2020

mattklein123 commented Jan 27, 2020

stale bot commented Feb 3, 2020

stale bot commented Feb 10, 2020

moderation Jan 22, 2020 •

edited

Loading

moderation Jan 25, 2020 •

edited

Loading

adamkotwasinski Jan 25, 2020 •

edited

Loading

moderation Jan 25, 2020 •

edited

Loading