panicked at 'already borrowed: BorrowMutError' in fetch

[jonmmease]

Hi, I'm the maintainer of vl-convert, which is a Rust library that depends on deno_runtime and deno_core to evaluate the Vega JavaScript visualization library to perform static image export. Deno has been awesome for this use-case, so thanks for making it possible to embed as a Rust library!

After updating deno_runtime to 0.117.0 and deno_core to 0.191.0 (or the current latest on master), I've started seeing a non-deterministic panic when operating under a heavy load.

thread 'tokio-runtime-worker' panicked at 'already borrowed: BorrowMutError', /Users/jonmmease/.cargo/git/checkouts/deno-22855de1c03c9128/aec761f/ext/fetch/lib.rs:461:6
stack backtrace:
   0:        0x16848774c - _rustsecp256k1_v0_8_1_default_error_callback_fn
   1:        0x168072588 - _BrotliDecoderVersion
   2:        0x168465bbc - _rustsecp256k1_v0_8_1_default_error_callback_fn
   3:        0x16848b67c - _rustsecp256k1_v0_8_1_default_error_callback_fn
   4:        0x16848b2dc - _rustsecp256k1_v0_8_1_default_error_callback_fn
   5:        0x16848c1c8 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   6:        0x16848bd78 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   7:        0x16848bce8 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   8:        0x16848bcdc - _rustsecp256k1_v0_8_1_default_error_callback_fn
   9:        0x1695b8aa8 - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  10:        0x1695b8cf4 - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  11:        0x168106260 - _BrotliDecoderVersion
  12:        0x16810e4f8 - _BrotliDecoderVersion
  13:        0x1680a03ec - _BrotliDecoderVersion
  14:        0x1680b77a0 - _BrotliDecoderVersion
  15:        0x1684b6474 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  16:        0x1684b4f50 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  17:        0x1684c2c04 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  18:        0x1684ba1c8 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  19:        0x1684b9db0 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  20:        0x1684bacd0 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  21:        0x16848df78 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  22:        0x19c227fa8 - __pthread_joiner_wake
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(61), ...)', /Users/jonmmease/.cargo/git/checkouts/deno-22855de1c03c9128/aec761f/core/runtime/jsruntime.rs:2267:50
stack backtrace:
   0:        0x16848774c - _rustsecp256k1_v0_8_1_default_error_callback_fn
   1:        0x168072588 - _BrotliDecoderVersion
   2:        0x168465bbc - _rustsecp256k1_v0_8_1_default_error_callback_fn
   3:        0x16848b67c - _rustsecp256k1_v0_8_1_default_error_callback_fn
   4:        0x16848b2dc - _rustsecp256k1_v0_8_1_default_error_callback_fn
   5:        0x16848c1c8 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   6:        0x16848bd78 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   7:        0x16848bce8 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   8:        0x16848bcdc - _rustsecp256k1_v0_8_1_default_error_callback_fn
   9:        0x1695b8aa8 - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  10:        0x1695b8cf4 - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  11:        0x1680aa050 - _BrotliDecoderVersion
  12:        0x16853f080 - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  13:        0x168540664 - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  14:        0x16853a4e8 - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  15:        0x16854cfbc - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  16:        0x16848df78 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  17:        0x19c227fa8 - __pthread_joiner_wake

If this is not a known issue, I can work on making a repro. But this may take a fair bit bit of time, so I wanted to check with you all first. Thanks!

[bartlomieju]

Hey, thanks for the report, we haven't seen this one yet. If you could come up with a repro that would definitely be helpful 👍

[jonmmease]

Ok, I made a repro in https://github.com/jonmmease/deno_19670_repro. I stripped down vl-convert as far as I could while preserving the error.

Let me know if there's anything else I can do to help!

[bartlomieju]

Thanks for the repro 👍 I'll try to fix it in the comint days.

[jonmmease]

Hi @bartlomieju, were you able to run the repro and see the error? Do you have any initial thoughts on how complex this will be to fix? It's not a rush, I'm just debating whether to downgrade Deno in vl-convert. Thanks again!

[bartlomieju]

I haven't had the chance yet. I'll try to reproduce it tonight.

[bartlomieju]

@jonmmease ah I think I found the problem - you are using multi-threaded Tokio runtime. With Deno you should use current-thread runtime - for performance reasons we did a hack suggested by the Tokio team that allows !Send futures to be used, but it's one of the invariants that we hold up internally that everything happens on the same thread.

[jonmmease]

That makes total sense, and fixes the issue. Thanks so much for taking the time to look into this for us!

[bartlomieju]

No worries, thanks for the report!