Flow to Kafka

[codefromthecrypt]

Recent versions of zipkin accept json encoding in kafka messages. I wonder if we can add a feature to write directly to kafka while flow'ing.

https://github.com/adrianco/spigo/blob/master/tooling/flow/flow.go

One option could be to use go-zipkin as opposed to, or as an alternative to writing json. It would be neat as it could make this a tracing load generator!
https://github.com/elodina/go-zipkin/blob/master/kafka.go

[codefromthecrypt]

@elodina are you interested enough to help with this?

[hubayirp]

+1 on both

On Mon, May 2, 2016 at 7:48 PM Adrian Cole [email protected] wrote:

Recent versions of zipkin accept json encoding in kafka messages. I wonder
if we can add a feature to write directly to kafka while flow'ing.

https://github.com/adrianco/spigo/blob/master/tooling/flow/flow.go

One option could be to use go-zipkin as opposed to, or as an alternative
to writing json. It would be neat as it could make this a tracing load
generator!
https://github.com/elodina/go-zipkin/blob/master/kafka.go

@elondina are you interested enough to help with this?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#67

[adrianco]

I looked at go-zipkin before and it makes too many assumptions about running on real systems I think. All we need is the basic interface wrappers for Kafka, as I already make the zipkin objects.

[codefromthecrypt]

zipkin's kafka support accepts the same json as POST, so yeah this should be doable

[codefromthecrypt]

@basvanbeek does this look like something you might be able to help with (plumb in the ability to write directly to kafka (or http) in addition to writing to a file?)

[basvanbeek]

I might be. Not sure yet when I have time for this though.

[adrianco]

How hard is it to get Kafka itself in place to test against?

On Wednesday, May 25, 2016, Bas van Beek [email protected] wrote:

I might be. Not sure yet when I have time for this though.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#67 (comment)

[codefromthecrypt]

Here's a paste of notes from here:
https://github.com/openzipkin/zipkin-java/tree/master/zipkin-server#kafka-collector

$ export KAFKA_ZOOKEEPER=$(docker-machine ip docker-machine active)#
Run Kafka in the background
$ docker run -d -p 2181:2181 -p 9092:9092
--env ADVERTISED_HOST=$KAFKA_ZOOKEEPER
--env AUTO_CREATE_TOPICS=true
spotify/kafka# Start the zipkin server, which reads $KAFKA_ZOOKEEPER
$ java -jar ./zipkin-server/target/zipkin-server-*exec.jar

On Wed, May 25, 2016 at 10:47 PM, Adrian Cockcroft <[email protected]

wrote:

How hard is it to get Kafka itself in place to test against?

On Wednesday, May 25, 2016, Bas van Beek [email protected] wrote:

I might be. Not sure yet when I have time for this though.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#67 (comment)

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#67 (comment)

[basvanbeek]

I created a docker-compose file with Zipkin + Zookeeper + Kafka + Cassandra which can be found here:

https://github.com/go-kit/kit/blob/master/tracing/zipkin/_docker/docker-compose-zipkin.yml

you can start it like this:

HOSTNAME=localhost docker-compose -f docker-compose-zipkin.yml up

[hubayirp]

Getting Kafka in place is not the hard part though..

On Wed, May 25, 2016 at 9:47 PM Adrian Cockcroft [email protected]
wrote:

How hard is it to get Kafka itself in place to test against?

On Wednesday, May 25, 2016, Bas van Beek [email protected] wrote:

I might be. Not sure yet when I have time for this though.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#67 (comment)

—
You are receiving this because you commented.

Reply to this email directly or view it on GitHub
#67 (comment)

[codefromthecrypt]

hard or not.. here's a kafka file that seems to work well enough openzipkin-attic/docker-zipkin#105

[hubayirp]

👍🏽
On Fri, Jul 22, 2016 at 1:48 AM Adrian Cole [email protected]
wrote:

hard or not.. here's a kafka file that seems to work well enough
openzipkin-attic/docker-zipkin#105
openzipkin-attic/docker-zipkin#105

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#67 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACWGWHFVPuiCLwbCQOOfwUU6FQa3S4ecks5qYIPqgaJpZM4IV453
.