Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

zipkin aws storage elasticsearch not working with aws elsticsearch version 7+ #196

Closed
ax-korneenkov opened this issue Jan 21, 2022 · 6 comments
Closed

zipkin aws storage elasticsearch not working with aws elsticsearch version 7+ #196

ax-korneenkov opened this issue Jan 21, 2022 · 6 comments
Labels
bug

Comments

@ax-korneenkov
Copy link

I am using elasticsearch 7.10 image on aws. When I connect to elasticsearch I use the openzipkin/zipkin-aws:0.23 image. and i get an error:

DEBUG 1 --- [orker-epoll-2-2] z.s.i.ZipkinHttpCollector                : Cannot decode spans due to IllegalArgumentException(Expected value at line 1 column 1 path $ reading List<Span> from json)

java.lang.IllegalArgumentException: Expected value at line 1 column 1 path $ reading List<Span> from json
	at zipkin2.internal.JsonCodec.exceptionReading(JsonCodec.java:230) ~[zipkin-2.21.7.jar:?]
	at zipkin2.internal.JsonCodec.readList(JsonCodec.java:151) ~[zipkin-2.21.7.jar:?]
	at zipkin2.codec.SpanBytesDecoder$3.decodeList(SpanBytesDecoder.java:127) ~[zipkin-2.21.7.jar:?]
	at zipkin2.codec.SpanBytesDecoder.doDecodeList(SpanBytesDecoder.java:209) ~[zipkin-2.21.7.jar:?]
	at zipkin2.codec.SpanBytesDecoder$3.decodeList(SpanBytesDecoder.java:143) ~[zipkin-2.21.7.jar:?]
	at zipkin2.collector.Collector.acceptSpans(Collector.java:149) ~[zipkin-collector-2.21.7.jar:?]
	at zipkin2.server.internal.ZipkinHttpCollector.lambda$validateAndStoreSpans$0(ZipkinHttpCollector.java:154) ~[classes/:?]
	at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) [?:?]
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) [?:?]
	at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) [?:?]
	at java.util.concurrent.CompletableFuture.complete(Unknown Source) [?:?]
	at com.linecorp.armeria.common.HttpMessageAggregator.onComplete(HttpMessageAggregator.java:92) [armeria-0.99.9.jar:?]
	at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:269) [armeria-0.99.9.jar:?]
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent0(DefaultStreamMessage.java:264) [armeria-0.99.9.jar:?]
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent(DefaultStreamMessage.java:256) [armeria-0.99.9.jar:?]
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.handleCloseEvent(DefaultStreamMessage.java:436) [armeria-0.99.9.jar:?]
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:368) [armeria-0.99.9.jar:?]
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber(DefaultStreamMessage.java:315) [armeria-0.99.9.jar:?]
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.addObjectOrEvent(DefaultStreamMessage.java:301) [armeria-0.99.9.jar:?]
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.close(DefaultStreamMessage.java:443) [armeria-0.99.9.jar:?]
	at com.linecorp.armeria.server.Http1RequestDecoder.channelRead(Http1RequestDecoder.java:253) [armeria-0.99.9.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152) [netty-handler-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.51.Final.jar:4.1.51.Final]
	at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: zipkin2.internal.gson.stream.MalformedJsonException: Expected value at line 1 column 1 path $
	at zipkin2.internal.gson.stream.JsonReader.syntaxError(JsonReader.java:1564) ~[zipkin-2.21.7.jar:?]
	at zipkin2.internal.gson.stream.JsonReader.doPeek(JsonReader.java:591) ~[zipkin-2.21.7.jar:?]
	at zipkin2.internal.gson.stream.JsonReader.beginArray(JsonReader.java:344) ~[zipkin-2.21.7.jar:?]
	at zipkin2.internal.JsonCodec$JsonReader.beginArray(JsonCodec.java:55) ~[zipkin-2.21.7.jar:?]
	at zipkin2.internal.JsonCodec.readList(JsonCodec.java:145) ~[zipkin-2.21.7.jar:?]

Such problems were not noticed with the use of elasticsearch aws version 6.8

I suspect that this may be due to the fact that in elasticsearch 7+ Indices prior to 7.0 could contain a colon (:), but that’s been deprecated and won’t be supported in 7.0+
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html
@jcchavezs
Copy link
Contributor

jcchavezs commented Jan 21, 2022 via email

@xeraa
Copy link
Member

xeraa commented Jan 21, 2022

  1. Yes, you cannot create indices with a colon in 7.0+ any more but you can still read from it.
  2. There was an earlier discussion on it in Determine how to address the next Elasticsearch index pattern break zipkin#2219 and I thought this was generally fixed? Also 7.0 came out like 2 years ago, so there must be something special / uncommon going on here.

@xf0e
Copy link

xf0e commented Feb 10, 2023

Hello,
we can't upgrade our ELK-Cluter from version 6 to 7 due to this ":"-restriction. Is there any chance to receive an update for this issue. Idealy with a fix, so zipkin cat create an Index in Elastic >= 7.0?

@shakuzen
Copy link
Member

@xf0e As @xeraa mentioned, it's not a limitation in Zipkin; it is a restriction in Elasticsearch that an index cannot be created with : from Elasticsearch 7. You will have to use a different delimiter. As far as I remember from the outcome of the issue @xeraa linked, Zipkin should use a different delimiter when writing to Elasticsearch >= 7.0.

@shakuzen
Copy link
Member

@ax-korneenkov that stacktrace is showing that an exception is being thrown by the collector, not the storage layer. It is failing to decode spans sent to the Zipkin collector. Make sure that spans are being sent in the correct format to the correct endpoint.

@codefromthecrypt
Copy link
Member

I think @shakuzen was correct. maybe this is the wrong input data. in any case, if this were an issue it should be on the zipkin repo, which has been significantly updated recently

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@codefromthecrypt @xeraa @xf0e @jcchavezs @shakuzen @ax-korneenkov