WIP: Elasticsearch uses Span2 internally

[codefromthecrypt]

This uses the span2 model from #1499 to implement Elasticsearch without using nested queries. This requires code from #1673

[codefromthecrypt]

added a "shared" field to help with status quo where libraries on the client and server side usually share a span ID. #1499 (comment)

Made one of the converters, but lacking tests at the moment. Once there's 2-way conversion, I'll open another PR trying out ES or similar

[codefromthecrypt]

lack of tests was driving me bonkers. Backfilled tests so that next change (which includes conversion code) is more sane

[codefromthecrypt]

I've a working span converter, but still needs a lot more tests as there are many ways data can come in (and come in unexpectedly)

[codefromthecrypt]

to move this forward, I'm first temporarily converting the internals of InMemorySpanStore to use SimpleSpan (and the converter). This will help break tests and show bugs before trying in elasticsearch.

[codefromthecrypt]

ok have almost all storage tests working.. should finish it tomorrow and move to ES

[codefromthecrypt]

update: all spanstore tests work, but some dependency linker tests don't probably missing some edge cases

[codefromthecrypt]

ok almost there on the data conversion. a couple more tests to go

[codefromthecrypt]

all tests are good with local span store.

I moved onto @openzipkin/elasticsearch, but ran into a glitch and need feedback on the idea or reverting to span.timestamp/duration (so that duration query is effortless) #1499 (comment)

[codefromthecrypt]

tentatively changed from startTimestamp/finishTimestamp to timestamp/duration to see if elasticsearch becomes easier (almost certainly will)

[codefromthecrypt]

@openzipkin/elasticsearch I added a commit for elasticsearch which uses the new format internally (no change to the format collectors use). Can I get a guinea pig to test this out by building this branch and throwing traffic at it? Bear in mind it requires ES 2.4+

[codefromthecrypt]

ps thanks @ImFlog and @anuraaga for the help getting ES to work. Would have been a lot harder without you!

[codefromthecrypt]

FYI working on dependency link now. deferring ES as I expect that it might be effortful for folks to test a branch

[codefromthecrypt]

dependency linking (killing DependencyLinkSpan for SimpleSpan) went well enough except one-way spans. I might have to re-jig the converters a bit to keep span store tests passing and also pass on dependency links. Glad I hit a snag as this stuff really needs to be solid.

[codefromthecrypt]

cleanups all done and provably we won't need DependencyLinkSpan after this. We can use the same span type for input, storage, query and aggregation. To that end, I've raised a comment whether to rename SimpleSpan to zipkin2.Span.

#1499 (comment)

Will proceed in whichever direction following a couple people feeding back. Regardless, it will be chopping this PR into smaller pieces.

[jcarres-mdsol]

What's the timeline? First changes to internal storage then API or all together?

[codefromthecrypt]

Here's a thought about order First is adding the type SimpleSpan or v2 span whatever we decide and converters. Then add the json serialisation code In any order we add content type handler for post, kafka etc transports Also in any order we add ElasticSearch support (storage format). We can add this as optional flag especially as indexing needs to be looked at carefully. All of the above can be done pretty quickly once we make a decision on naming, but I would probably aim for elasticsearch even before transports because there are a lot of wins there.

…

On 27 Jul 2017 5:53 am, "Jordi Polo Carres" ***@***.***> wrote: What's the timeline? First changes to internal storage then API or all together? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1651 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD617t5ul8HK71Svx5IWNWZmMTyuqtkks5sR7U8gaJpZM4OVQvn> .

[codefromthecrypt]

So everything mentioned including ES internal in next release is a proposal. Next minor from that would improve or formalize that and add transport (http kafka) support.

…

On 27 Jul 2017 8:42 am, "Adrian Cole" ***@***.***> wrote: Here's a thought about order First is adding the type SimpleSpan or v2 span whatever we decide and converters. Then add the json serialisation code In any order we add content type handler for post, kafka etc transports Also in any order we add ElasticSearch support (storage format). We can add this as optional flag especially as indexing needs to be looked at carefully. All of the above can be done pretty quickly once we make a decision on naming, but I would probably aim for elasticsearch even before transports because there are a lot of wins there. On 27 Jul 2017 5:53 am, "Jordi Polo Carres" ***@***.***> wrote: > What's the timeline? First changes to internal storage then API or all > together? > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#1651 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAD617t5ul8HK71Svx5IWNWZmMTyuqtkks5sR7U8gaJpZM4OVQvn> > . >

[jcarres-mdsol]

The flag is a good idea, would allow for incremental adoption. I would imagine that we just wipe out old data and store new in the new format. I think most users do not store the data for long anyways

[codefromthecrypt]

@jcarres-mdsol I think I found a way to tease out a lot of the code safely.

If I start with swapping the internal DependencyLinkSpan for an internal copy of what's now SimpleSpan, we can get that code out there and make sure it works. Later will be a simple import change to formalize the type.

[codefromthecrypt]

first thing splitting off is the span model type (no codec or converters) #1669

[codefromthecrypt]

All the rejigging PRs are green. I plan to merge the ones leading up to this (Elasticsearch spike) tomorrow. If folks are keen on the new span kinds for producer/consumer, I'll add a PR for that.

Once the above is done, I'll look at how hard it is to make this have an internal flag to use the "span2" type conditionally (right now, this branch won't read old data only new data). If it is straightforward enough, I'll push that out with a minor bump adding instructions here on how to enable it.

In the mean time, anyone can try this branch on their own. It writes to the same indexes, but using the "span2" type and no "servicespan" index.

[codefromthecrypt]

heh.. something's up as github won't let me re-open this

[codefromthecrypt]

carried forward in #1674