Drive trans from the output of the translation item collector #33890

michaelwoerister · 2016-05-26T19:43:45Z

This PR changes the way how translation works above the item level. Instead of walking the HIR and calling trans_item() on everything encountered (while instantiating monomorphizations on-demand), we now just process the list of translation items generated by the trans::collector. Using the collector has the benefit of being able to know the exact set of monomorphizations and symbols before actually running translation, something that is crucial for incremental compilation (but also has other benefits).

The collector has existed for quite a while now, but so far it's output was only used for running some auto-tests. With this PR it becomes the only source of truth about what gets translated.

One modification we had to make, compared to the initial approach, is that closures are not represented as their own TransItems. Doing so, while still supporting non-MIR-based translation, would have been prohibitively complex, and not worth the trouble since legacy-trans will disappear sooner or later. Once there is solely MIR-trans, it would be a good idea to make closures TransItems again.

This PR removes the most obvious functions and tables that are not needed anymore, but there's definitely still more cleanup possible later on (e.g. monomorphize::monomorphic_fn() does very little at this point). Since there are already more than 10 commits in here, doing this in a separate PR seems to be a better idea.

These changes definitely warrant a crater run.

Thanks @Aatch, for taking on one of the more tedious tasks during the dev-sprint!
Thanks @eddyb, for doing some nice refactorings to symbol name generation and making sure these landed so I could use them!

cc @rust-lang/compiler
cc @rust-lang/tools

rust-highfive · 2016-05-26T19:43:58Z

r? @Aatch

(rust_highfive has picked a reviewer for you, use r? to override)

eddyb · 2016-05-26T19:49:38Z

src/librustc_trans/base.rs

-                    }
-                }
-            }
-        }
        hir::ItemEnum(ref enum_definition, ref gens) => {
            if gens.ty_params.is_empty() {
                // sizes only make sense for non-generic types
                enum_variant_size_lint(ccx, enum_definition, item.span, item.id);


This should definitely be a regular lint using ty::layout now.

Preexisting, though.

Right, mostly making a note as it's the only thing I can see left in that function.

nikomatsakis · 2016-05-26T20:49:27Z

r? @nikomatsakis

(or @eddyb ?)

bors · 2016-05-26T21:28:56Z

☔ The latest upstream changes (presumably #33783) made this pull request unmergeable. Please resolve the merge conflicts.

michaelwoerister · 2016-05-27T14:29:47Z

Rebased...

bors · 2016-05-30T23:36:37Z

☔ The latest upstream changes (presumably #33909) made this pull request unmergeable. Please resolve the merge conflicts.

nikomatsakis · 2016-05-31T20:22:06Z

src/librustc_trans/monomorphize.rs

        }) => {
            attributes::from_fn_attrs(ccx, attrs, lldecl);
-
-            let is_first = !ccx.available_monomorphizations().borrow()
-                                                             .contains(&symbol);


so...was this code just dead?

AFAICT, this logic was moved elsewhere.

Yes, the partitioner is now responsible for assigning the correct linkage to monomorphizations.

nikomatsakis · 2016-05-31T22:28:54Z

OK, so, I gave this a once over and it all makes sense. I left various comments. r=me but I'd like @eddyb to sign off too.

r? @eddyb

nikomatsakis · 2016-06-01T01:26:47Z

Kicked off a crater run, but it occurs to me that a more likely source of failure is Windows and the like.

eddyb · 2016-06-01T03:54:59Z

LGTM, other than @nikomatsakis' comments.

nikomatsakis · 2016-06-01T08:27:24Z

@michaelwoerister crater run yielded 8 regressions. I looked at a few of them and they looked legit:

https://gist.github.com/nikomatsakis/4b1496d1b60e77cac2fe8c604a6ebec2

michaelwoerister · 2016-06-01T15:55:22Z

Thanks for the crater run! Hopefully the regressions can be fixed quickly.

michaelwoerister · 2016-06-02T18:38:04Z

So, if been looking into the first regression from the crater run and it seems that MIR is sufficiently different from old-trans that it breaks. The culprit here is the following function:

fn exists<P: AsRef<Path>>(path: P) -> io::Result<bool> {
    match fs::metadata(path) {
        Ok(_) => Ok(true),
        Err(ref err) if err.kind() == io::ErrorKind::NotFound => Ok(false),
        Err(err) => Err(err)
    }
}

When compiling with -Zorbit everything is fine, but when compiling with old-trans there is a linker error because old-trans expects a monomorphization for exists<&&Path> but there is only one for exists<&PathBuf>. Does anyone have an idea how that could happen? Does lowering to MIR do something different with auto-deref maybe?

jonas-schievink · 2016-06-02T18:45:22Z

@michaelwoerister This also seems to break a test when compiling with CGU (works fine without): https://gist.github.com/jonas-schievink/ad3fbdc66029a287f73ca9d2f3229027

(this is with make check-stage1-rpass RUSTFLAGS="-Ccodegen-units=4" TESTNAME="associated-types-normalize-unifield-struct")

…uiting logical operations.

…nd monomorphizations. See issue rust-lang#34151 for more information.

… them.

The data tracked here was meant to compare the output of the translation item collector to the set of translation items found by the on-demand translator.

michaelwoerister · 2016-07-08T14:48:20Z

@bors r=eddyb

Re-added a missing use statement that had been removed in the meantime.

bors · 2016-07-08T14:48:21Z

📌 Commit 1c03bfe has been approved by eddyb

bors · 2016-07-08T15:34:37Z

⌛ Testing commit 1c03bfe with merge d119362...

@Aatch

Drive trans from the output of the translation item collector This PR changes the way how translation works above the item level. Instead of walking the HIR and calling `trans_item()` on everything encountered (while instantiating monomorphizations on-demand), we now just process the list of translation items generated by the `trans::collector`. Using the collector has the benefit of being able to know the exact set of monomorphizations and symbols before actually running translation, something that is crucial for incremental compilation (but also has [other benefits](#33602)). The collector has existed for quite a while now, but so far it's output was only used for running some auto-tests. With this PR it becomes the only source of truth about what gets translated. One modification we had to make, compared to the initial approach, is that closures are not represented as their own `TransItems`. Doing so, while still supporting non-MIR-based translation, would have been prohibitively complex, and not worth the trouble since legacy-trans will disappear sooner or later. Once there is solely MIR-trans, it would be a good idea to make closures `TransItems` again. This PR removes the most obvious functions and tables that are not needed anymore, but there's definitely still more cleanup possible later on (e.g. `monomorphize::monomorphic_fn()` does very little at this point). Since there are already more than 10 commits in here, doing this in a separate PR seems to be a better idea. These changes definitely warrant a crater run. Thanks @Aatch, for taking on one of the more tedious tasks during the dev-sprint! Thanks @eddyb, for doing some nice refactorings to symbol name generation and making sure these landed so I could use them! cc @rust-lang/compiler cc @rust-lang/tools

bors · 2016-07-08T22:00:06Z

…s, r=compiler-errors Fix codegen-units tests that were disabled 8 years ago I don't know if any of these tests still have value. They were disabled by rust-lang#33890, and we've survived without them for a while. But considering how small this test suite is, maybe it's worth having them. I also had to add some normalization to the codegen-units tests output. I think the fact that I had to add some underscores how poor our test coverage is.

Rollup merge of rust-lang#128929 - saethlin:enable-codegen-units-tests, r=compiler-errors Fix codegen-units tests that were disabled 8 years ago I don't know if any of these tests still have value. They were disabled by rust-lang#33890, and we've survived without them for a while. But considering how small this test suite is, maybe it's worth having them. I also had to add some normalization to the codegen-units tests output. I think the fact that I had to add some underscores how poor our test coverage is.

rust-highfive assigned Aatch May 26, 2016

eddyb reviewed May 26, 2016
View reviewed changes

michaelwoerister unassigned Aatch May 26, 2016

rust-highfive assigned nikomatsakis May 26, 2016

michaelwoerister force-pushed the collector-driven-trans branch from f972fbb to 7e3b445 Compare May 27, 2016 14:27

nikomatsakis reviewed May 31, 2016
View reviewed changes

rust-highfive assigned eddyb and unassigned nikomatsakis May 31, 2016

michaelwoerister force-pushed the collector-driven-trans branch from 7e3b445 to 91e9bda Compare June 2, 2016 15:32

michaelwoerister added 10 commits July 8, 2016 10:42

Pacify make tidy.

00226fc

collector-driven-trans: Take care of nits.

ab80f74

trans::collector: Also consider initializers of const items.

b33240e

hir-trans: Don't generate code for unreachable operands in short-circ…

4a3f9b8

…uiting logical operations.

trans: Add missing normalize_associated_type() call to callee::get_fn().

a7bc0b9

trans: Enable falling back to on-demand instantiation for drop-glue a…

4c27a3c

…nd monomorphizations. See issue rust-lang#34151 for more information.

trans: Set COMDAT section for weak symbols so that Windows can handle…

b149b9d

… them.

trans: Remove tracking of translation item state.

ac80d41

The data tracked here was meant to compare the output of the translation item collector to the set of translation items found by the on-demand translator.

Update LLVM.

051d391

trans: Adjust linkage assignment so that we don't need weak linkage.

1c03bfe

michaelwoerister force-pushed the collector-driven-trans branch from 64a9f96 to 1c03bfe Compare July 8, 2016 14:44

bors merged commit 1c03bfe into rust-lang:master Jul 8, 2016

bors mentioned this pull request Jul 8, 2016

Use ptr::{null, null_mut} instead of 0 as *{const, mut} #34456

Merged

tomaka mentioned this pull request Jul 10, 2016

Regression: linking errors since latest nightly #34754

Closed

This was referenced Jul 12, 2016

rustup to 2016-07-11 servo/servo#12393

Closed

trans: Avoid using weak_odr linkage for closures. #34793

Closed

jonas-schievink mentioned this pull request Jul 17, 2016

Performance regression in nightly #34891

Closed

nagisa mentioned this pull request Jul 17, 2016

Apparent 1000% regression in llvm phase of bootstrap #34831

Closed

michaelwoerister mentioned this pull request Sep 14, 2016

codegen-units setting in Cargo.toml causes LLVM error with nightly-i686-pc-windows-msvc #36309

Closed

saethlin mentioned this pull request Aug 10, 2024

Fix codegen-units tests that were disabled 8 years ago #128929

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Drive trans from the output of the translation item collector #33890

Drive trans from the output of the translation item collector #33890

michaelwoerister commented May 26, 2016

rust-highfive commented May 26, 2016

eddyb May 26, 2016

nagisa May 26, 2016

eddyb May 26, 2016

nikomatsakis commented May 26, 2016

bors commented May 26, 2016

michaelwoerister commented May 27, 2016

bors commented May 30, 2016

nikomatsakis May 31, 2016

eddyb Jun 1, 2016

michaelwoerister Jun 1, 2016

nikomatsakis commented May 31, 2016

nikomatsakis commented Jun 1, 2016

eddyb commented Jun 1, 2016

nikomatsakis commented Jun 1, 2016

michaelwoerister commented Jun 1, 2016

michaelwoerister commented Jun 2, 2016

jonas-schievink commented Jun 2, 2016

michaelwoerister commented Jul 8, 2016

bors commented Jul 8, 2016

bors commented Jul 8, 2016

bors commented Jul 8, 2016

Drive trans from the output of the translation item collector #33890

Drive trans from the output of the translation item collector #33890

Conversation

michaelwoerister commented May 26, 2016

rust-highfive commented May 26, 2016

eddyb May 26, 2016

Choose a reason for hiding this comment

nagisa May 26, 2016

Choose a reason for hiding this comment

eddyb May 26, 2016

Choose a reason for hiding this comment

nikomatsakis commented May 26, 2016

bors commented May 26, 2016

michaelwoerister commented May 27, 2016

bors commented May 30, 2016

nikomatsakis May 31, 2016

Choose a reason for hiding this comment

eddyb Jun 1, 2016

Choose a reason for hiding this comment

michaelwoerister Jun 1, 2016

Choose a reason for hiding this comment

nikomatsakis commented May 31, 2016

nikomatsakis commented Jun 1, 2016

eddyb commented Jun 1, 2016

nikomatsakis commented Jun 1, 2016

michaelwoerister commented Jun 1, 2016

michaelwoerister commented Jun 2, 2016

jonas-schievink commented Jun 2, 2016

michaelwoerister commented Jul 8, 2016

bors commented Jul 8, 2016

bors commented Jul 8, 2016

bors commented Jul 8, 2016