Windows builds fail to link C++ static library

[jdm]

Code

Building the mozjs crate for any Windows target stopped working after #97485. mozjs has a build script that compiles the C++ library SpiderMonkey into js_static.a, which is then linked. Before #97485, these builds worked successfully. Since that PR was merged, CI using nightly and beta toolsets targeting MSVC x86_64 and UWP x86_64 now fail to build with the following error:

error: failed to add native library D:\a\mozjs\mozjs\target\debug\build\mozjs_sys-4062d5cd7cbc7110\out\build/js/src/build\js_static.lib: Unsupported archive identifier

This was first reported in servo/mozjs#327 and servo/mozjs#334.

Version it worked on

2022-12-03 nightly
Rust 1.66 beta

Version with regression

2022-12-04 nightly
Rust 1.67 beta

If you know when this regression occurred, please add a line like below, replacing {channel} with one of stable, beta, or nightly.

@rustbot modify labels: +regression-from-stable-to-beta -regression-untriaged

[jyn514]

cc @bjorn3

[bjorn3]

As workaround adding windows to

rust/compiler/rustc_codegen_llvm/src/back/archive.rs

Line 105 in 85da15c

if sess.target.arch == "wasm32" || sess.target.arch == "wasm64" {

should revert back to the old archive writer. Luckily it was preserved for wasm.

Could you post a hex dump of the first 16 or so bytes of the js_static.lib archive to see the magic bytes of this file.

[sagudev]

Here is hexdump:

0000000 3c21 6874 6e69 0a3e 202f 2020 2020 2020
0000010 2020 2020 2020 2020 2030 2020 2020 2020

Here is GitHub action of working one: https://github.com/sagudev/mozjs/actions/runs/3974783209 and here is GitHub action of defect one: https://github.com/sagudev/mozjs/actions/runs/3974683394. Artifact contains .lib file, that is produced in windows job.

[bjorn3]

Yeah, that is indeed very much not a valid archive.

┌────────┬─────────────────────────┬────────┐
│00000000│ 3c 21 68 74 6e 69 0a 3e │<!htni_>│
│00000008│ 20 2f 20 20 20 20 20 20 │ /      │
│00000010│ 20 20 20 20 20 20 20 20 │        │
│00000018│ 20 30 20 20 20 20 20 20 │ 0      │
└────────┴─────────────────────────┴────────┘

It does look like !<thin>\n (which is a valid header for thin archives) except with every pair of bytes swapped. Rustc also shouldn't be producing thin archives.

[sagudev]

Well hex-editor in vscode shows it right (but hexdump is showing reversed bits on same file):

[mati865]

Do you know if this affects all Windows targets or just MSVC based?
I couldn't get it to build at all with windows-gnu:

$ LANG=C.utf8 cargo t
   Compiling mozjs_sys v0.68.2 (D:\tmp\mozjs\mozjs)
error: failed to run custom build command for `mozjs_sys v0.68.2 (D:\tmp\mozjs\mozjs)`

Caused by:
  process didn't exit successfully: `D:\tmp\mozjs\target\debug\build\mozjs_sys-352d87b28c39ffac\build-script-build` (exit code: 101)
  --- stdout
  cargo:outdir=D:\tmp\mozjs\target\debug\build\mozjs_sys-27c0928751302e7b\out\build

  --- stderr
  make: invalid --jobserver-auth string '__rust_jobserver_semaphore_2019855554'
  make: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
  D:\tmp\mozjs\mozjs\makefile.cargo:175: *** recipe commences before first target.  Stop.
  thread 'main' panicked at 'assertion failed: result.success()', mozjs\build.rs:178:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[sagudev]

Ideally we would want to find another case for regression, because mozjs is complicated.
@mati865 Do you have MozillaBuild 3.4 installed as error is in makefile.cargo:175 and MozBuild comes packaged with python? I suggest looking at GitHub Workflows to correctly setup for compiling.

[mati865]

@mati865 Do you have MozillaBuild 3.4 installed as error is in makefile.cargo:175 and MozBuild comes packaged with python?

No, I had checked first step in the readme but the link is dead:

Follow the directions at https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Windows_Prerequisites

[sagudev]

Well, those instruction would not be relevant anymore as we are still few SM versions behind if they would be available. Following GitHub Workflows is currently the best way.

[apiraino]

WG-prioritization assigning priority (Zulip discussion).

@rustbot label -I-prioritize +P-high

[pnkfelix]

@rustbot label: -P-high +P-critical

(Upgrading priority because, until we better understand the scope of this bug, it strikes me as something that might motivate a fast-tracked point release for 1.67.)

[apiraino]

I wonder if we have another case of this issue in #107162

[bjorn3]

You linked this issue itself.

[lqd]

Could be #107334

[bjorn3]

Oh, I completely misunderstood what the issue was. I thought this was about rustc producing a malformed archive, but it is actually about rustc not being able to read thin archives produced by an external build system. Object indeed doesn't support parsing thin archives and even if it did, it would likely need to be wired up in rustc separately.

I think the best course of action is to change new_archive_writer to always take the

rust/compiler/rustc_codegen_llvm/src/back/archive.rs

Line 106 in c62665e

Box::new(LlvmArchiveBuilder { sess, additions: Vec::new() })

branch. This will restore the old behavior for cg_llvm. #105221 will probably also need to be reverted partially or as a whole because it removed macOS fat archive reading for the LLVM archive writer.

(This would have been so much more painful to revert if I hadn't needed to keep the old LLVM archive writer for wasm.)

[bjorn3]

And yes #107334 is very likely a duplicate of this.

[kanerogers]

I believe I may be encountering a similar, or related, regression between 1.67 and 1.66 when attempting to build and link against shaderc on MSVC: google/shaderc-rs#133.

[bjorn3]

Opened #107360 to revert back to LlvmArchiveWriter, which would fix this issue for now I think.

[pnkfelix]

@rustbot label: E-needs-test

[pnkfelix]

(and any such test should probably be beta-backported; see discussion from T-compiler triage meeting)

[wesleywiser]

Downgrading to P-medium as we've shipped a point release with this fix. Leaving open as we would love a regression test to be added.

[Mark-Simulacrum]

Untagging as a regression, since this has been fixed on beta + stable + nightly at this point as far as I can tell.