TimeoutException with Latest Build

[neilchaudhuri]

With the latest build of play-redis, I had an error on Heroku that emanated from play-redis. It could be an issue with Brando, but I am curious why that Await call happens:

java.util.concurrent.TimeoutException: Futures timed out after [1 second]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:169)
at scala.concurrent.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3640)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:167)
at scala.concurrent.Await$.result(package.scala:190)
at play.api.cache.redis.Implicits$Synchronizer.sync(Implicits.scala:38)
at play.api.cache.redis.Builders$SynchronousBuilder$.toResult(Builders.scala:50)
at play.api.cache.redis.Implicits$class.build(Implicits.scala:43)
at play.api.cache.redis.RedisCache.build(RedisCache.scala:17)
at play.api.cache.redis.RedisCache.get(RedisCache.scala:32)
at play.api.cache.redis.SyncRedis.get(Cache.scala:12)
at play.api.cache.CachedBuilder$$anonfun$build$1$$anonfun$apply$7.apply(Cached.scala:197)
at play.api.cache.CachedBuilder$$anonfun$build$1$$anonfun$apply$7.apply(Cached.scala:197)
at scala.Option.orElse(Option.scala:289)
at play.api.cache.CachedBuilder$$anonfun$build$1.apply(Cached.scala:195)
at play.api.cache.CachedBuilder$$anonfun$build$1.apply(Cached.scala:184)
at play.api.mvc.EssentialAction$$anon$2.apply(Action.scala:55)
at play.api.mvc.EssentialAction$$anon$2.apply(Action.scala:54)
at play.core.routing.TaggingInvoker$$anon$2.apply(HandlerInvoker.scala:37)
at play.core.routing.TaggingInvoker$$anon$2.apply(HandlerInvoker.scala:36)
at play.filters.gzip.GzipFilter$$anon$1.apply(GzipFilter.scala:54)
at play.filters.gzip.GzipFilter$$anon$1.apply(GzipFilter.scala:51)
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$9.apply(PlayDefaultUpstreamHandler.scala:155)

Here is what I saw in the logs:

a.a.OneForOneStrategy - queue empty 
java.util.NoSuchElementException: queue empty 
    at scala.collection.mutable.Queue.dequeue(Queue.scala:66) ~[org.scala-lang.scala-library-2.11.7.jar:na] 
    at brando.Connection$$anonfun$receive$1$$anonfun$applyOrElse$4.apply(Connection.scala:91) ~[com.digital-achiever.brando_2.11-3.0.3.jar:3.0.3]  
    at brando.Connection$$anonfun$receive$1$$anonfun$applyOrElse$4.apply(Connection.scala:78) ~[com.digital-achiever.brando_2.11-3.0.3.jar:3.0.3]  
    at brando.ReplyParser$class.parseReply(ReplyParser.scala:145) ~[com.digital-achiever.brando_2.11-3.0.3.jar:3.0.3] 
    at brando.Connection.parseReply(Connection.scala:27) ~[com.digital-achiever.brando_2.11-3.0.3.jar:3.0.3] 
    at brando.Connection$$anonfun$receive$1.applyOrElse(Connection.scala:78) ~[com.digital-achiever.brando_2.11-3.0.3.jar:3.0.3] 
    at akka.actor.Actor$class.aroundReceive(Actor.scala:467) ~[com.typesafe.akka.akka-actor_2.11-2.3.13.jar:na] 
    at brando.Connection.aroundReceive(Connection.scala:27) ~[com.digital-achiever.brando_2.11-3.0.3.jar:3.0.3] 
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) [com.typesafe.akka.akka-actor_2.11-2.3.13.jar:na] 
    at akka.actor.ActorCell.invoke(ActorCell.scala:487) [com.typesafe.akka.akka-actor_2.11-2.3.13.jar:na]

Any ideas what the issue is? Perhaps a bug in the latest version of Brando?

[neilchaudhuri]

I should note that I did clear the cache manually. I don't know if that is relevant.

Here is the exception from the start:

[error] p.a.cache.redis - GET command failed for key 'homepage'. 
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://application/user/$b#1174980798]] after [1000 ms] 
    at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334) ~[com.typesafe.akka.akka-actor_2.11-2.3.13.jar:na] 
    at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) ~[com.typesafe.akka.akka-actor_2.11-2.3.13.jar:na] 
    at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599) ~[org.scala-lang.scala-library-2.11.7.jar:na] 
    at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) ~[org.scala-lang.scala-library-2.11.7.jar:na] 
    at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597) ~[org.scala-lang.scala-library-2.11.7.jar:na] 
    at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467) ~[com.typesafe.akka.akka-actor_2.11-2.3.13.jar:na] 
    at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419) ~[com.typesafe.akka.akka-actor_2.11-2.3.13.jar:na] 
    at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423) ~[com.typesafe.akka.akka-actor_2.11-2.3.13.jar:na] 
    at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375) ~[com.typesafe.akka.akka-actor_2.11-2.3.13.jar:na] 
    at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66-cedar14] 
[info] p.a.cache.redis - Redis cache started. Actor is connected to [redis-host] 

 [error] a.a.OneForOneStrategy - queue empty 
 java.util.NoSuchElementException: queue empty

[KarelCemus]

I am curious why that Await call happens

This is called by the library because you are using synchronous API. The Brando connector is asynchronous but the Play's cache API is not. To overcome this it uses the Await. The timeout can be configured in the conf file

[KarelCemus]

Otherwise I don't know what happens there, maybe Brando devs might help. Their core is not documented and I'm not sure what the faulty code is supposed to do.

[neilchaudhuri]

Ahh, the Await makes sense then. Meanwhile, I have posted already on the Brando board but have yet to hear anything.

Unfortunately, this error came out of nowhere and brings down the entire application on startup. SO I have had to disable play-redis and rely on Play's default. It makes me sad--particularly since the error has so little explanation.

[KarelCemus]

I've been thinking how to help you when I am unable to track down the bug. I am about to release this version as 1.0.0 when its tested (please provide me some feedback at the end of the year, I'm not using it right now so I'm not sure it's working). What if I release the previous version with brando 2.0 but current api as 0.3? Would it help you? That means there would be a stable release and we could independently test the version with Brando 3.x.x.

Thoughts?

[KarelCemus]

Actually, I could release it as 1.0.0-M1 or 1.0.0-RC1 which is stable but only a milestone or release candidate, respectively. Then I can release another RC and finally 1.0.0. I think this is good idea.

Thoughs?

[neilchaudhuri]

I think your milestone and release candidate ideas are good ideas.

[KarelCemus]

Please check out releases.

[KarelCemus]

Cannot reproduce and it seems like Brando issue. Closing for now.

[KarelCemus]

Duplicate of #44
Narrowed down as Brando #67