Error: response body closed before all bytes were read

[sentialx]

xwin download throws Error: response body closed before all bytes were read on Linux/macOS making cross-compilation effectively not working.
On Windows it's working.

[jjohannes]

It looks like this is flaky. I ran this on both Linux (GitHub Actions runner) and MacOS (my local machine) many times the past two days without ever encountering this problem. Today, it fails almost all the time on the GitHub runner (but it also works sometimes).

Error: response body closed before all bytes were read

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from

   1: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
   2: rayon::iter::plumbing::Folder::consume_iter
   3: rayon::iter::plumbing::bridge_producer_consumer::helper
   4: rayon_core::join::join_context::{{closure}}
   5: rayon::iter::plumbing::bridge_producer_consumer::helper
   6: rayon_core::job::StackJob<L,F,R>::run_inline
   7: rayon_core::join::join_context::{{closure}}
   8: rayon::iter::plumbing::bridge_producer_consumer::helper
   9: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
  10: rayon_core::registry::WorkerThread::wait_until_cold
  11: rayon_core::registry::ThreadBuilder::run
  12: std::sys::backtrace::__rust_begin_short_backtrace
  13: core::ops::function::FnOnce::call_once{{vtable.shim}}
  14: std::sys::pal::unix::thread::Thread::new::thread_start
  15: <unknown>
  16: <unknown>

[Jake-Shadle]

Try https://github.com/Jake-Shadle/xwin/releases/tag/0.6.6-rc.1 and see if you have issues still, I realize it's flaky so it won't be 100% certain, but this is my current best guess at a fix.

[jjohannes]

Thank you for providing a fix so quickly @Jake-Shadle. Will let you know if I still encounter the issue with 0.6.6-rc.1.

[jjohannes]

Unfortunately, the issue occurs (for me).

[Jake-Shadle]

Ok, in that case, try doing RAYON_NUM_THREADS=1 xwin ... when executing so that it only ever uses one thread to see if that avoids (or possibly hides) the issue. I can't repro this so it's a bit hard to know what exactly is wrong, so right now I'm assuming it is something with request parallelism.

[jjohannes]

Unfortunately setting RAYON_NUM_THREADS=1 does not solve it. Unfortunately, I have no idea how I could help reproducing the problem. I also never got it on my Mac, just sometimes on Linux GH agents.

[Jake-Shadle]

I can't see your CI job, is it the exact same error? That should be impossible, the original error comes from ureq.

[sentialx]

@Jake-Shadle Looks like the maturin devs (PyO3/maturin#2222) are not willing to upgrade to latest xwin as it's using "bloated reqwest" as a dependency. Maybe ureq v3 could solve this?

[Jake-Shadle]

Well the move to reqwest didn't fix it, I doubt using ureq 3 will either, something else is going on at a deeper level.

[sentialx]

Maybe Microsoft servers are closing the connection? I was able to workaround the issue by retrying xwin download until it works. Maybe a retry mechanism is needed here?

[jjohannes]

I can't see your CI job, is it the exact same error? That should be impossible, the original error comes from ureq.

Sorry my bad. I forgot that the repo is not yet public. I removed the links.
I will give it another try soon and post an update.

[Jake-Shadle]

I suppose it's possible, but it could also be some kind of rate limiting happening in Github actions, is it happening for you on a local machine as well? I've literally never seen this locally nor in this project's CI, but if it happens only in Github actions it would indicate they are doing something, not Microsoft. Well, I guess they're both Microsoft, but not the SDK servers.

[sentialx]

It is happening for me locally both on Linux (Docker) and macOS. Windows works well.

[Jake-Shadle]

Ok, maybe it is a server issue then, I'll add retries.

[jjohannes]

Alright, using RAYON_NUM_THREADS=1 I get this one:

Error: response body closed before all bytes were read
Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
   1: xwin::ctx::Ctx::get_and_validate
   2: xwin::manifest::get_manifest
   3: xwin::main
   4: std::sys::backtrace::__rust_begin_short_backtrace
   5: std::rt::lang_start::{{closure}}
   6: std::rt::lang_start_internal
   7: main
   8: <unknown>
   9: __libc_start_main
  10: _start

Hope that is helpful. This is with v0.6.5. Testing with the RC now.

[jjohannes]

With 0.6.6-rc.1 and RAYON_NUM_THREADS=1 it works for me. At least it worked three times in a row now while it was failing quite constantly before.

Thanks a lot for the support @Jake-Shadle.