Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cargo Raze ignores binary dependencies #218

Closed
UebelAndre opened this issue Sep 15, 2020 · 10 comments
Closed

Cargo Raze ignores binary dependencies #218

UebelAndre opened this issue Sep 15, 2020 · 10 comments

Comments

@UebelAndre
Copy link
Collaborator

UebelAndre commented Sep 15, 2020

I was doing some tests trying to refresh things in rules_rust and came across an issue with generating targets for @io_bazel_rules_rust//wasm_bindgen/raze/....

There appears to have been a regression introduced in 816ea69 (merged by #134) where a lot of the targets are no longer being generated. I've yet to nail down the specific cause. But, if you run cargo raze on or after this commit, you get the following output.

Loaded override settings: RazeSettings {
    workspace_path: "//wasm_bindgen/raze",
    target: "x86_64-unknown-linux-gnu",
    crates: {
        "syn": {
            "0.15.44": CrateSettings {
                additional_deps: [],
                skipped_deps: [],
                extra_aliased_targets: [],
                additional_flags: [
                    "--cfg=syn_can_call_macro_by_path",
                ],
                gen_buildrs: true,
                data_attr: None,
                buildrs_additional_environment_variables: [],
                patch_args: [],
                patch_cmds: [],
                patch_cmds_win: [],
                patch_tool: None,
                patches: [],
            },
        },
        "proc-macro2": {
            "0.4.30": CrateSettings {
                additional_deps: [],
                skipped_deps: [],
                extra_aliased_targets: [],
                additional_flags: [
                    "--cfg=use_proc_macro",
                ],
                gen_buildrs: false,
                data_attr: None,
                buildrs_additional_environment_variables: [],
                patch_args: [],
                patch_cmds: [],
                patch_cmds_win: [],
                patch_tool: None,
                patches: [],
            },
        },
        "wasm-bindgen": {
            "0.2.48": CrateSettings {
                additional_deps: [],
                skipped_deps: [],
                extra_aliased_targets: [
                    "cargo_bin_wasm_bindgen",
                ],
                additional_flags: [],
                gen_buildrs: true,
                data_attr: None,
                buildrs_additional_environment_variables: [],
                patch_args: [],
                patch_cmds: [],
                patch_cmds_win: [],
                patch_tool: None,
                patches: [],
            },
        },
    },
    gen_workspace_prefix: "raze",
    genmode: Remote,
    output_buildfile_suffix: "BUILD",
}
Generated remote/BUILD successfully
Generated .//remote/bumpalo-2.6.0.BUILD successfully
Generated .//remote/cfg-if-0.1.10.BUILD successfully
Generated .//remote/lazy_static-1.4.0.BUILD successfully
Generated .//remote/log-0.4.11.BUILD successfully
Generated .//remote/proc-macro2-0.4.30.BUILD successfully
Generated .//remote/quote-0.6.13.BUILD successfully
Generated .//remote/syn-0.15.44.BUILD successfully
Generated .//remote/unicode-xid-0.1.0.BUILD successfully
Generated .//remote/wasm-bindgen-0.2.48.BUILD successfully
Generated .//remote/wasm-bindgen-backend-0.2.48.BUILD successfully
Generated .//remote/wasm-bindgen-macro-0.2.48.BUILD successfully
Generated .//remote/wasm-bindgen-macro-support-0.2.48.BUILD successfully
Generated .//remote/wasm-bindgen-shared-0.2.48.BUILD successfully
Generated .//BUILD successfully
Generated .//crates.bzl successfully

This however, is a much smaller list than what is already generated. If you go back one commit before the one I mentioned, you get a list similar to what was originally generated back in October 2019 (see bazelbuild/rules_rust#240)

Loaded override settings: RazeSettings {
    workspace_path: "//wasm_bindgen/raze",
    target: "x86_64-unknown-linux-gnu",
    crates: {
        "wasm-bindgen": {
            "0.2.48": CrateSettings {
                additional_deps: [],
                skipped_deps: [],
                extra_aliased_targets: [
                    "cargo_bin_wasm_bindgen",
                ],
                additional_flags: [],
                gen_buildrs: true,
                data_attr: None,
                buildrs_additional_environment_variables: [],
                patch_args: [],
                patch_cmds: [],
                patch_cmds_win: [],
                patch_tool: None,
                patches: [],
            },
        },
        "proc-macro2": {
            "0.4.30": CrateSettings {
                additional_deps: [],
                skipped_deps: [],
                extra_aliased_targets: [],
                additional_flags: [
                    "--cfg=use_proc_macro",
                ],
                gen_buildrs: false,
                data_attr: None,
                buildrs_additional_environment_variables: [],
                patch_args: [],
                patch_cmds: [],
                patch_cmds_win: [],
                patch_tool: None,
                patches: [],
            },
        },
        "syn": {
            "0.15.44": CrateSettings {
                additional_deps: [],
                skipped_deps: [],
                extra_aliased_targets: [],
                additional_flags: [
                    "--cfg=syn_can_call_macro_by_path",
                ],
                gen_buildrs: true,
                data_attr: None,
                buildrs_additional_environment_variables: [],
                patch_args: [],
                patch_cmds: [],
                patch_cmds_win: [],
                patch_tool: None,
                patches: [],
            },
        },
    },
    gen_workspace_prefix: "raze",
    genmode: Remote,
    output_buildfile_suffix: "BUILD",
}
Generated remote/BUILD successfully
Generated .//remote/addr2line-0.13.0.BUILD successfully
Generated .//remote/adler-0.2.3.BUILD successfully
Generated .//remote/aho-corasick-0.7.13.BUILD successfully
Generated .//remote/arrayref-0.3.6.BUILD successfully
Generated .//remote/arrayvec-0.5.1.BUILD successfully
Generated .//remote/ascii-0.8.7.BUILD successfully
Generated .//remote/atty-0.2.14.BUILD successfully
Generated .//remote/autocfg-0.1.7.BUILD successfully
Generated .//remote/autocfg-1.0.1.BUILD successfully
Generated .//remote/backtrace-0.3.50.BUILD successfully
Generated .//remote/base64-0.12.3.BUILD successfully
Generated .//remote/base64-0.9.3.BUILD successfully
Generated .//remote/bitflags-1.2.1.BUILD successfully
Generated .//remote/blake2b_simd-0.5.10.BUILD successfully
Generated .//remote/buf_redux-0.8.4.BUILD successfully
Generated .//remote/bumpalo-2.6.0.BUILD successfully
Generated .//remote/byteorder-1.3.4.BUILD successfully
Generated .//remote/cc-1.0.59.BUILD successfully
Generated .//remote/cfg-if-0.1.10.BUILD successfully
Generated .//remote/chrono-0.4.15.BUILD successfully
Generated .//remote/chunked_transfer-0.3.1.BUILD successfully
Generated .//remote/cloudabi-0.0.3.BUILD successfully
Generated .//remote/constant_time_eq-0.1.5.BUILD successfully
Generated .//remote/crossbeam-channel-0.4.4.BUILD successfully
Generated .//remote/crossbeam-deque-0.7.3.BUILD successfully
Generated .//remote/crossbeam-epoch-0.8.2.BUILD successfully
Generated .//remote/crossbeam-utils-0.7.2.BUILD successfully
Generated .//remote/curl-0.4.33.BUILD successfully
Generated .//remote/curl-sys-0.4.36+curl-7.71.1.BUILD successfully
Generated .//remote/dirs-1.0.5.BUILD successfully
Generated .//remote/docopt-1.1.0.BUILD successfully
Generated .//remote/either-1.6.0.BUILD successfully
Generated .//remote/env_logger-0.6.2.BUILD successfully
Generated .//remote/failure-0.1.8.BUILD successfully
Generated .//remote/failure_derive-0.1.8.BUILD successfully
Generated .//remote/filetime-0.2.12.BUILD successfully
Generated .//remote/fuchsia-cprng-0.1.1.BUILD successfully
Generated .//remote/getrandom-0.1.15.BUILD successfully
Generated .//remote/gimli-0.22.0.BUILD successfully
Generated .//remote/heck-0.3.1.BUILD successfully
Generated .//remote/hermit-abi-0.1.15.BUILD successfully
Generated .//remote/httparse-1.3.4.BUILD successfully
Generated .//remote/humantime-1.3.0.BUILD successfully
Generated .//remote/id-arena-2.2.1.BUILD successfully
Generated .//remote/idna-0.1.5.BUILD successfully
Generated .//remote/itoa-0.4.6.BUILD successfully
Generated .//remote/lazy_static-1.4.0.BUILD successfully
Generated .//remote/leb128-0.2.4.BUILD successfully
Generated .//remote/libc-0.2.77.BUILD successfully
Generated .//remote/libz-sys-1.1.2.BUILD successfully
Generated .//remote/log-0.3.9.BUILD successfully
Generated .//remote/log-0.4.11.BUILD successfully
Generated .//remote/matches-0.1.8.BUILD successfully
Generated .//remote/maybe-uninit-2.0.0.BUILD successfully
Generated .//remote/memchr-2.3.3.BUILD successfully
Generated .//remote/memoffset-0.5.5.BUILD successfully
Generated .//remote/mime-0.2.6.BUILD successfully
Generated .//remote/mime_guess-1.8.8.BUILD successfully
Generated .//remote/miniz_oxide-0.4.2.BUILD successfully
Generated .//remote/multipart-0.15.4.BUILD successfully
Generated .//remote/num-integer-0.1.43.BUILD successfully
Generated .//remote/num-traits-0.2.12.BUILD successfully
Generated .//remote/num_cpus-1.13.0.BUILD successfully
Generated .//remote/object-0.20.0.BUILD successfully
Generated .//remote/openssl-probe-0.1.2.BUILD successfully
Generated .//remote/openssl-sys-0.9.58.BUILD successfully
Generated .//remote/percent-encoding-1.0.1.BUILD successfully
Generated .//remote/phf-0.7.24.BUILD successfully
Generated .//remote/phf_codegen-0.7.24.BUILD successfully
Generated .//remote/phf_generator-0.7.24.BUILD successfully
Generated .//remote/phf_shared-0.7.24.BUILD successfully
Generated .//remote/pkg-config-0.3.18.BUILD successfully
Generated .//remote/ppv-lite86-0.2.9.BUILD successfully
Generated .//remote/proc-macro2-0.4.30.BUILD successfully
Generated .//remote/proc-macro2-1.0.21.BUILD successfully
Generated .//remote/quick-error-1.2.3.BUILD successfully
Generated .//remote/quote-0.6.13.BUILD successfully
Generated .//remote/quote-1.0.7.BUILD successfully
Generated .//remote/rand-0.4.6.BUILD successfully
Generated .//remote/rand-0.5.6.BUILD successfully
Generated .//remote/rand-0.6.5.BUILD successfully
Generated .//remote/rand-0.7.3.BUILD successfully
Generated .//remote/rand_chacha-0.1.1.BUILD successfully
Generated .//remote/rand_chacha-0.2.2.BUILD successfully
Generated .//remote/rand_core-0.3.1.BUILD successfully
Generated .//remote/rand_core-0.4.2.BUILD successfully
Generated .//remote/rand_core-0.5.1.BUILD successfully
Generated .//remote/rand_hc-0.1.0.BUILD successfully
Generated .//remote/rand_hc-0.2.0.BUILD successfully
Generated .//remote/rand_isaac-0.1.1.BUILD successfully
Generated .//remote/rand_jitter-0.1.4.BUILD successfully
Generated .//remote/rand_os-0.1.3.BUILD successfully
Generated .//remote/rand_pcg-0.1.2.BUILD successfully
Generated .//remote/rand_xorshift-0.1.1.BUILD successfully
Generated .//remote/rayon-1.4.0.BUILD successfully
Generated .//remote/rayon-core-1.8.0.BUILD successfully
Generated .//remote/rdrand-0.4.0.BUILD successfully
Generated .//remote/redox_syscall-0.1.57.BUILD successfully
Generated .//remote/redox_users-0.3.5.BUILD successfully
Generated .//remote/regex-1.3.9.BUILD successfully
Generated .//remote/regex-syntax-0.6.18.BUILD successfully
Generated .//remote/remove_dir_all-0.5.3.BUILD successfully
Generated .//remote/rouille-3.0.0.BUILD successfully
Generated .//remote/rust-argon2-0.8.2.BUILD successfully
Generated .//remote/rustc-demangle-0.1.16.BUILD successfully
Generated .//remote/ryu-1.0.5.BUILD successfully
Generated .//remote/safemem-0.3.3.BUILD successfully
Generated .//remote/schannel-0.1.19.BUILD successfully
Generated .//remote/scopeguard-1.1.0.BUILD successfully
Generated .//remote/serde-1.0.116.BUILD successfully
Generated .//remote/serde_derive-1.0.116.BUILD successfully
Generated .//remote/serde_json-1.0.57.BUILD successfully
Generated .//remote/sha1-0.6.0.BUILD successfully
Generated .//remote/siphasher-0.2.3.BUILD successfully
Generated .//remote/socket2-0.3.15.BUILD successfully
Generated .//remote/strsim-0.9.3.BUILD successfully
Generated .//remote/syn-0.15.44.BUILD successfully
Generated .//remote/syn-1.0.41.BUILD successfully
Generated .//remote/synstructure-0.12.4.BUILD successfully
Generated .//remote/tempdir-0.3.7.BUILD successfully
Generated .//remote/tempfile-3.1.0.BUILD successfully
Generated .//remote/term-0.5.2.BUILD successfully
Generated .//remote/termcolor-1.1.0.BUILD successfully
Generated .//remote/thread_local-1.0.1.BUILD successfully
Generated .//remote/threadpool-1.8.1.BUILD successfully
Generated .//remote/time-0.1.44.BUILD successfully
Generated .//remote/tiny_http-0.6.2.BUILD successfully
Generated .//remote/tinyvec-0.3.4.BUILD successfully
Generated .//remote/twoway-0.1.8.BUILD successfully
Generated .//remote/unicase-1.4.2.BUILD successfully
Generated .//remote/unicode-bidi-0.3.4.BUILD successfully
Generated .//remote/unicode-normalization-0.1.13.BUILD successfully
Generated .//remote/unicode-segmentation-1.6.0.BUILD successfully
Generated .//remote/unicode-xid-0.1.0.BUILD successfully
Generated .//remote/unicode-xid-0.2.1.BUILD successfully
Generated .//remote/url-1.7.2.BUILD successfully
Generated .//remote/vcpkg-0.2.10.BUILD successfully
Generated .//remote/version_check-0.1.5.BUILD successfully
Generated .//remote/walrus-0.8.0.BUILD successfully
Generated .//remote/walrus-macro-0.8.0.BUILD successfully
Generated .//remote/wasi-0.10.0+wasi-snapshot-preview1.BUILD successfully
Generated .//remote/wasi-0.9.0+wasi-snapshot-preview1.BUILD successfully
Generated .//remote/wasm-bindgen-0.2.48.BUILD successfully
Generated .//remote/wasm-bindgen-anyref-xform-0.2.48.BUILD successfully
Generated .//remote/wasm-bindgen-backend-0.2.48.BUILD successfully
Generated .//remote/wasm-bindgen-cli-0.2.48.BUILD successfully
Generated .//remote/wasm-bindgen-cli-support-0.2.48.BUILD successfully
Generated .//remote/wasm-bindgen-macro-0.2.48.BUILD successfully
Generated .//remote/wasm-bindgen-macro-support-0.2.48.BUILD successfully
Generated .//remote/wasm-bindgen-shared-0.2.48.BUILD successfully
Generated .//remote/wasm-bindgen-threads-xform-0.2.48.BUILD successfully
Generated .//remote/wasm-bindgen-wasm-interpreter-0.2.48.BUILD successfully
Generated .//remote/wasm-webidl-bindings-0.1.2.BUILD successfully
Generated .//remote/wasmparser-0.30.0.BUILD successfully
Generated .//remote/winapi-0.3.9.BUILD successfully
Generated .//remote/winapi-i686-pc-windows-gnu-0.4.0.BUILD successfully
Generated .//remote/winapi-util-0.1.5.BUILD successfully
Generated .//remote/winapi-x86_64-pc-windows-gnu-0.4.0.BUILD successfully
Generated .//BUILD successfully
Generated .//crates.bzl successfully
@UebelAndre
Copy link
Collaborator Author

@acmcarther While you're around, maybe you could dump some context on this issue and the change associated with it?

@UebelAndre
Copy link
Collaborator Author

@damienmg you might want to check this out as well.

@UebelAndre UebelAndre changed the title Raze is not generating all the data it used to Cargo Raze ignores binary dependencies Sep 16, 2020
@UebelAndre
Copy link
Collaborator Author

I started a thread on zulip and I think the issue has been identified. It seems cargo metadata filters out binary dependencies. This means that adding wasm-bindgen-cli as a dependency to your project will do nothing. I don't think this should be the case but maybe @damienmg @acmcarther @GregBowyer @smklein @mfarrugi might have some suggestions here.

@UebelAndre
Copy link
Collaborator Author

I'm wondering now if there should be a new raze section for binary dependencies based on the zulip conversation.

@UebelAndre
Copy link
Collaborator Author

This appears to be an issue in cargo itself: rust-lang/cargo#8708

@UebelAndre
Copy link
Collaborator Author

I think until binary dependencies are officially supported by cargo that maybe there should be a dedicated raze field to handle them.

@UebelAndre
Copy link
Collaborator Author

@acmcarther I could really use your input here. Is this already a known and solved issue?

@acmcarther
Copy link
Member

acmcarther commented Sep 17, 2020

I don't want to set an expectation that I always appear when asked (I am really flaky), but I'll try to answer questions that I have the context for.


Bummer about this situation. Aight so the backstory:

Originally, the implementation of this crate linked cargo as a dependency directly and used its internals to resolve the complete set of dependencies. With that, we could identify the BUILD files that needed to be generated and generate them.

It turned out that having cargo as a dependency represented a headache for users, particularly those that vendored cargo-raze, because cargo itself has a huge number of dependencies, and never really represented an API that users should be depending on.

To that end, at some point in the past, Cargo got a "cargo-metadata" command which purported to emit all of the metadata that one would need in order to construct the full dependency tree and what not. I refactored the raze implementation in order to support using either cargo internals, or "cargo-metadata".

In practice, the folks harmed by the ongoing dependency that this crate had on cargo (the crate) were not well served until that cargo internals implementation could be deleted. So #134 represented making the "cargo-metadata" implementation the default, then some later PR removed the cargo internals implementation.

I had a general awareness that there might be issues here which were not obvious because they would only show up for users with esoteric use cases. It's surprising to me to read that so many dependencies are omitted.


You've linked above a relevant cargo issue. That probably represents the root cause. Basically, cargo-metadata deigns to give us only a subset of the data which was originally available from cargo's own internals.

I don't have a good idea of what the next steps should be. There's always the option of adding an optional dependency on the cargo internals again and supporting that as an implementation again. Beyond that though I really don't know the options, especially as they pertain to changes within cargo itself.

@UebelAndre
Copy link
Collaborator Author

@acmcarther Hey! Thanks so much for your reply! I hope I'm not over stepping by pinging everyone. I asked for you in particular because I had a feeling you would have context on this 😃.

I have an idea for explicitly handling binary dependencies which I'm hoping to show in a PR soon. Maybe I'll be able to finish it over the weekend. Other than that, maybe cargo_internal could be added again behind a feature flag but I feel like that would not be the greatest since it would reintroduce old issues and feels kinda like relying on undefined behavior. But even then maybe it's the smoothest implementation.

@UebelAndre
Copy link
Collaborator Author

This is resolved as of #227 and was released in 0.6.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants