Mutex relocking error when a node went down

[iconara]

Looks like the CqlProtocolHandler lock is held when a callback is called, and in certain situations (like when a connection fails) something in the call chain of the callback reads metadata from the protocol handler, which tries to take the lock.

ThreadError: Mutex relocking by same thread
  org/jruby/ext/thread/Mutex.java:90:in `lock'
  org/jruby/ext/thread/Mutex.java:147:in `synchronize'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/protocol/cql_protocol_handler.rb:64:in `[]'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/client/asynchronous_prepared_statement.rb:28:in `execute'
  (application frame)
  org/jruby/RubyProc.java:255:in `call'
  (application frame)
  (application frame)
  org/jruby/RubyProc.java:255:in `call'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/future.rb:228:in `fallback'
  org/jruby/RubyProc.java:255:in `call'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/future.rb:402:in `fail'
  org/jruby/RubyArray.java:1617:in `each'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/future.rb:401:in `fail'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/future.rb:41:in `fail'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/future.rb:141:in `map'
  org/jruby/RubyProc.java:255:in `call'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/future.rb:402:in `fail'
  org/jruby/RubyArray.java:1617:in `each'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/future.rb:401:in `fail'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/future.rb:41:in `fail'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/protocol/cql_protocol_handler.rb:251:in `socket_closed'
  org/jruby/RubyArray.java:1617:in `each'
  org/jruby/RubyEnumerable.java:920:in `each_with_index'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/protocol/cql_protocol_handler.rb:249:in `socket_closed'
  org/jruby/ext/thread/Mutex.java:149:in `synchronize'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/protocol/cql_protocol_handler.rb:248:in `socket_closed'
  org/jruby/RubyMethod.java:134:in `call'
  org/jruby/RubyProc.java:255:in `call'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/io/connection.rb:216:in `closed!'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/io/connection.rb:76:in `close'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/io/connection.rb:184:in `read'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/io/connection.rb:182:in `read'
  org/jruby/RubyArray.java:1617:in `each'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/io/io_reactor.rb:329:in `check_sockets!'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/io/io_reactor.rb:308:in `tick'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/cql-rb-1.1.0/lib/cql/io/io_reactor.rb:139:in `start'
  org/jruby/RubyProc.java:255:in `call'
  /usr/local/rvm/gems/jruby-1.7.4@xyz/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `create_with_logging_context'

[michaelklishin]

Switching to Monitor (require"monitor") will help.

[iconara]

In this case the right solution is to not hold the lock while calling the callbacks (it's completely unnecessary and was sloppy to begin with).

[iconara]

But in general I should probably use Monitor over Mutex.