pyoxidizer run failed on Windows

[lvscar]

Environment

• pyOxidizer : 0.2.0
• rustc: 1.36.0
• windows: win10 1809

When I run pyoxidizer build or pyoxidizer run on fresh inited pyoxidizer project, I got the following error:

writing packed Python module and resource data...
9727413 bytes of Python module data written to C:/code/tmp/testing_on_win\build\target\x86_64-pc-windows-msvc\debug\pyoxidizer\py-modules
4 bytes of resources data written to C:/code/tmp/testing_on_win\build\target\x86_64-pc-windows-msvc\debug\pyoxidizer\python-resources

generating custom link library containing Python...
deriving custom config.c from 83 extension modules
compiling custom config.c to object file
running: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.21.27702\\bin\\HostX64\\x64\\cl.exe" "/nologo" "/MD" "/I" "C:\\Users\\LVS~1.DES\\AppData\\Local\\Temp\\libpython.oe6Hx69Gv3xc" "/W4" "-std=c99" "/DNDEBUG" "/DPy_BUILD_CORE" "/FoC:/code/tmp/testing_on_win\\build\\target\\x86_64-pc-windows-msvc\\debug\\pyoxidizer\\config.o" "/c" "C:\\Users\\LVS~1.DES\\AppData\\Local\\Temp\\libpython.oe6Hx69Gv3xc\\config.c"
cargo:warning=cl: config.c
C:\Users\User\AppData\Local\Temp\libpython.oe6Hx69Gv3xc\pytime.h(123): warning C4115: “timeval”: 括号中的已命名类型
定义
thread 'main' panicked at 'failed printing to stdout: Windows stdio in console mode does not support writing non-UTF-8 byte sequences', src\libstd\io\stdio.rs:792:9
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

changing stdio_encoding value not help.

[indygreg]

Oh, fun.

The Windows stdio in console mode message appears to come from Rust: https://github.com/rust-lang/rust/blob/master/src/libstd/sys/windows/stdio.rs. It appears some more context about this is available at rust-lang/rust#23344.

Built PyOxidizer applications should not be going through Rust's I/O layer to write to stdout. And judging from your captured output, it kind of looks like the error was emitted as part of compiling the Rust program! I think Visual Studio emitted a Chinese localized warning message and when cargo/pyoxidizer captured the output (probably as [u8] and wrote it back to the console as-is, an error occurred in Rust's I/O layer because the byte stream wasn't exactly UTF-8.

We can work around this in PyOxidizer by setting environment variables to force English in Visual Studio output. But that's not a great user experience for non-English speakers. We could also normalize process output to UTF-8, using the standard replacement character when output doesn't conform to UTF-8. There may also be encoding settings that we can apply to the invoked processes so Windows does the conversion for us and the [u8] we receive as process output is valid UTF-8.

Could you please run set RUSTC_BACKTRACE=1 then try running the failed command again so we can see exactly where this is failing?

@luser do you have any insight to add?

[lapoigne]

Same problem,

I just installed rust on windows 10 and follow the pyoxidizer tutorial

Whem running pyoxidizer run I got the same error

BACKTRACE

writing packed Python module and resource data...
9727413 bytes of Python module data written to C:/Data/trav/python/archive2/pyapp\build\target\x86_64-pc-windows-msvc\debug\pyoxidizer\py-modules
4 bytes of resources data written to C:/Data/trav/python/archive2/pyapp\build\target\x86_64-pc-windows-msvc\debug\pyoxidizer\python-resources
generating custom link library containing Python...
deriving custom config.c from 83 extension modules
compiling custom config.c to object file
running: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX64\\x64\\cl.exe" "/nologo" "/MD" "/I" "C:\\Users\\vprelat\\AppData\\Local\\Temp\\libpython.y60z2tYxfAVx" "/W4" "-std=c99" "/DNDEBUG" "/DPy_BUILD_CORE" "/FoC:/Data/trav/python/archive2/pyapp\\build\\target\\x86_64-pc-windows-msvc\\debug\\pyoxidizer\\config.o" "/c" "C:\\Users\\vprelat\\AppData\\Local\\Temp\\libpython.y60z2tYxfAVx\\config.c"
cargo:warning=clconfig.c
c:\users\vprelat\appdata\local\temp\libpython.y60z2tyxfavx\pytime.h(123): warning C4115: 'timeval' : définition d'un type nommé dans les parenthèses
thread 'main' panicked at 'failed printing to stdout: Windows stdio in console mode does not support writing non-UTF-8 byte sequences', src\libstd\io\stdio.rs:792:9

stack backtrace:
   0: git_packbuilder_object_count
   1: git_packbuilder_object_count
   2: git_packbuilder_object_count
   3: git_packbuilder_object_count
   4: git_packbuilder_object_count
   5: git_packbuilder_object_count
   6: git_packbuilder_object_count
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>
  16: <unknown>
  17: <unknown>
  18: <unknown>
  19: <unknown>
  20: <unknown>
  21: <unknown>
  22: <unknown>
  23: <unknown>
  24: git_packbuilder_object_count
  25: git_packbuilder_object_count
  26: git_packbuilder_object_count
  27: <unknown>
  28: git_libgit2_version
  29: BaseThreadInitThunk
  30: RtlUserThreadStart

[luser]

When sccache runs msvc it explicitly converts its output from the local codepage:
https://github.com/mozilla/sccache/blob/6ba6f6c15c106768b914a7697a763e2232fa253a/src/compiler/msvc.rs#L154

Poking around the pyoxidizer source for the log messages from previous comments it looks like this call into the cc crate is to blame:

PyOxidizer/pyoxidizer/src/pyrepackager/repackage.rs

Line 1225 in 9208ea2

cc::Build::new()

and sure enough:
https://github.com/alexcrichton/cc-rs/blob/85bc5b9fce81177ed8024e5241136417c6008db8/src/lib.rs#L2377-L2413

That code will definitely fail in this situation on Windows! I'd recommend opening an issue about this on the cc repo. It should be feasible to have it do the right thing on Windows.

[indygreg]

Thank you so much @luser!

I agree with your assessment that this is a bug in the cc crate. I have just filed a new issue against that project (see link above this comment).

I have to admit that I'm relatively surprised this bug has lingered so long in the cc crate. I wonder if Rust doesn't have a very active non-English Windows community...

While I'm here, it is highly possible that PyOxidizer itself has occurrences of this does not normalize output to UTF-8 issue. I will likely need to audit the code base for every instance where we invoke a process and ensure we're doing something sane with text encodings. Also, it appears that Rust's standard library doesn't provide a way to access the OsString for process output? That seems a bit unfortunate. My recollection is that the stdio streams in Windows are wchar_t by default. I'm really curious what normalization, if any, Rust is doing to the raw streams. It feels like std::process::Command may not have the platform-specific nuances that primitives like std::path::Path have. That would be... unfortunate.

[lapoigne]

The workaround:
Run visual studio installer, modify the installation and install English language pack.
I have only French the first time and the VSLANG set doesn't work.

Run set VSLANG=1033 before pyoxydizer command
The compilation should be ok