At least once mode keeps requeueing commands on non-recoverable errors

[mrvisser]

For a brief period when getting set up with lettuce, I had a bogus dependency set up -- mixed versions of netty modules. That caused me to hit a NoSuchMethodError at a strange spot in a lettuce codec, which somehow ended up in a permanently locked event loop thread:

"lettuce-nioEventLoop-5-1" #21 daemon prio=5 os_prio=31 tid=0x00007fb2ac774800 nid=0x6107 runnable [0x0000700005939000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.Throwable.fillInStackTrace(Native Method)
    at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
    - locked <0x00000006c32fafd8> (a java.lang.NoSuchMethodError)
    at java.lang.Throwable.<init>(Throwable.java:265)
    at java.lang.Error.<init>(Error.java:70)
    at java.lang.LinkageError.<init>(LinkageError.java:55)
    at java.lang.IncompatibleClassChangeError.<init>(IncompatibleClassChangeError.java:55)
    at java.lang.NoSuchMethodError.<init>(NoSuchMethodError.java:58)
    at com.lambdaworks.redis.codec.StringCodec.estimateSize(StringCodec.java:127)
    at com.lambdaworks.redis.protocol.CommandArgs$KeyArgument.encode(CommandArgs.java:619)
    at com.lambdaworks.redis.protocol.CommandArgs.encode(CommandArgs.java:337)
    at com.lambdaworks.redis.protocol.Command.encode(Command.java:120)
    at com.lambdaworks.redis.protocol.AsyncCommand.encode(AsyncCommand.java:180)
    at com.lambdaworks.redis.protocol.CommandEncoder.encode(CommandEncoder.java:91)
    at com.lambdaworks.redis.protocol.CommandEncoder.encode(CommandEncoder.java:76)
    at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:107)
        ...

Here was the code I was running to try out lettuce that resulted in this hang-up:

object RedisMain extends App {
  println(Version.identify.get("netty-common"))

  val resources = DefaultClientResources.builder()
    .dnsResolver(DnsResolvers.JVM_DEFAULT)
    .build()
  val client = RedisClient.create(resources, "redis://localhost")
  val conn = client.connect()
  println(conn.sync().get("test")) // Hung on .get("test")
  conn.close()
  client.shutdown()
}

The netty-common dependency printed above was netty-common-4.1.13.Final.c7f8168 and, unfortunately, I do not have all the specific netty module versions that were being pulled in at this exact point, but some were also 4.1.13 and some (e.g., transport) around 4.0.x.

I would have expected the get invocation to throw in this case. It might be worth a check to ensure that there isn't a strange vector for exceptions resulting in locked IO threads -- but it's entirely possible this reaction is also a side-effect of invalid dependencies.

Thanks for lettuce!

[mp911de]

That's fixed now. The issue was caused by Lettuce's retry behavior. When sending a command runs on an error, commands are retried assuming the issue was caused by an I/O error. The retry now inspects the actual error and unrecoverable errors lead to command cancellation.