Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustbuild: Rewrite user-facing interface #37521

Merged
merged 1 commit into from
Nov 6, 2016

Conversation

alexcrichton
Copy link
Member

This commit is a rewrite of the user-facing interface to the rustbuild build
system. The intention here is to make it much easier to compile/test the project
without having to remember weird rule names and such. An overall view of the new
interface is:

# build everything
./x.py build

# document everyting
./x.py doc

# test everything
./x.py test

# test libstd
./x.py test src/libstd

# build libcore stage0
./x.py build src/libcore --stage 0

# run stage1 run-pass tests
./x.py test src/test/run-pass --stage 1

The src/bootstrap/bootstrap.py script is now aliased as a top-level x.py
script. This x was chosen to be both short and easily tab-completable (no
collisions in that namespace!). The build system now accepts a "subcommand" of
what to do next, the main ones being build/doc/test.

Each subcommand then receives an optional list of arguments. These arguments are
paths in the source repo of what to work with. That is, if you want to test a
directory, you just pass that directory as an argument.

The purpose of this rewrite is to do away with all of the arcane renames like
"rpass" is the "run-pass" suite, "cfail" is the "compile-fail" suite, etc. By
simply working with directories and files it's much more intuitive of how to run
a test (just pass it as an argument).

The rustbuild step/dependency management was also rewritten along the way to
make this easy to work with and define, but that's largely just a refactoring of
what was there before.

The intention is that this support is extended for arbitrary files (e.g.
src/test/run-pass/my-test-case.rs), but that isn't quite implemented just yet.
Instead directories work for now but we can follow up with stricter path
filtering logic to plumb through all the arguments.

@alexcrichton
Copy link
Member Author

cc @jonathandturner

@rust-highfive
Copy link
Collaborator

r? @brson

(rust_highfive has picked a reviewer for you, use r? to override)


./x.py build src/libcore
./x.py build src/libproc_macro
./x.py build src/libstd --stage 1 --filter hash_map
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does --filter do?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah this is actually a way to pass extra arguments to the test harness itself (e.g. the executable). In most cases that ends up being a filter, but not always in the case of things like -q or such.

I'll change the name of this to --test-args

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rename is good if it's really just arbitrary arguments to the test runner. But there's another confusing thing here: The example invokes build, not test. Does this ``build` command include testing or is that a typo?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops the --filter here was totally off, moved it elsewhere

@brson
Copy link
Contributor

brson commented Nov 1, 2016

Seems good, but I'll mostly take your word for it that it's correct.

I see rules are defined by a name and a path. Does that mean I can use either to invoke them? e.g. how do I run tidy? Is it reasonable to have a command that lists all the rules?

@alexcrichton
Copy link
Member Author

@brson the name of each rule is actually just a human readable string for defining rules itself. There's not necessarily a path for all rules so the path can't be the unique key. I figured it was just best to define everything in one place instead of scattered across a few.

And yeah so to run tidy you'd do:

./x.py test src/tools/tidy

I do think it's reasonable yeah to print out all options, I'll look for a location to put that.

@petrochenkov
Copy link
Contributor

@alexcrichton
What are equivalents of these commands now?

make rustc-stage1 -j8
make check-stage1-rustdocck -j8 TESTNAME=foo
make check-notidy -j8

Also, could you answer the questions in https://internals.rust-lang.org/t/the-rustbuild-feature-thread/3643/19 for the updated interface?

@alexcrichton
Copy link
Member Author

@brson ok, updated!

@petrochenkov

In rustbuild there's no need for -j as it automatically uses all parallelism you've got. You can override that with an extra -j flag to the x.py script thought.

Otherwise the equivalences are:

  • make rustc-stage => make rustc-stage. That is, the makefile generated still has this. Otherwise it's ./x.py build src/libtest --stage 1
  • make check-stage1-rustdocck TESTNAME=foo => ./x.py test src/test/rustdoc --test-args foo
  • make check-notidy => ./x.py test -- that is, there's no way to run all checks except tidy.

And sure! Once we've landed this I can respond to that thread.

@petrochenkov
Copy link
Contributor

@alexcrichton
Thanks!

In rustbuild there's no need for -j as it automatically uses all parallelism you've got

All is usually too much if you're doing some other work on the same machine, that's why I usually use a fixed number.

@brson
Copy link
Contributor

brson commented Nov 2, 2016

@bors r+

@bors
Copy link
Contributor

bors commented Nov 2, 2016

📌 Commit d760453 has been approved by brson

@brson
Copy link
Contributor

brson commented Nov 2, 2016

I suggest posting to irlo about this when it lands since it changes the entire interface.

@japaric
Copy link
Member

japaric commented Nov 2, 2016

Is python src/bootstrap/bootstrap.py --stage 1 --step libstd-link --target arm-unknown-linux-gnueabi (cross compile std for ARM using the stage1 rustc) ./x.py build --stage 1 --step libstd-link --target arm-unknown-linux-gnueabi now?

@alexcrichton
Copy link
Member Author

@japaric that invocation would now look like:

./x.py build --stage 1 src/libstd --host arm-unknown-linux-gnueabi --target arm-unknown-linux-gnueabi

That is, you want to create a stage1 host compiler which has the stage1 libstd library, all for the target arm-unknown-linux-gnueabi.

The libstd-link old step was pretty special, and was kinda wonky. If all you want is a standard library for a particular target then you can drop the --host argument

@japaric
Copy link
Member

japaric commented Nov 2, 2016

If all you want is a standard library for a particular target then you can drop the --host argument

Yes, that's what I meant. Will this produce a "proper" sysroot that's usable with rustup toolchain link? That is a sysroot that contains both libstd.rlib for x86_64 (host) and libstd.rlib for ARM (target). Because the python bootstrap.py command that I posted produced an incomplete sysroot with only libstd.rlib for ARM (target).

That is, you want to create a stage1 host compiler which has the stage1 libstd library, all for the target arm-unknown-linux-gnueabi.

Oh, that's nifty. I think there wasn't a command for that before. If there was, I have no idea what it was. 😄

bors added a commit that referenced this pull request Nov 2, 2016
Rollup of 10 pull requests

- Successful merges: #37351, #37405, #37473, #37482, #37488, #37498, #37502, #37513, #37517, #37523
- Failed merges: #37521
@alexcrichton
Copy link
Member Author

Ah yes so the method of multiple targets and specifying what to build was also updated to be "more correct" as part of this rewrite. That is, let's say you configured with one host of ARM and one target of powerpc. This means that x86_64 is the build triple, ARM and x86_64 are the host triples, and then there's three target triples. The various possibilities for building things are:

# Build every libstd possible. That is:
#   * arm/lib/x86_64/libstd
#   * arm/lib/arm/libstd
#   * arm/lib/powerpc/libstd
#   * x86_64/lib/x86_64/libstd
#   * x86_64/lib/arm/libstd
#   * x86_64/lib/powerpc/libstd
./x.py build src/libstd

# Build libstd for all hosts for one target:
#   * arm/lib/arm/libstd
#   * x86_64/lib/arm/libstd
./x.py build src/libstd --target arm

# Build libstd for one host and one target:
#   * arm/lib/arm/libstd
./x.py build src/libstd --target arm --host arm

I haven't personally used rustup toolchain link all that much, but from what I believe then what you'll want to do is to configure a bunch of targets, run ./x.py build src/libtest (e.g. up to the test harness), and then the sysroot should be ready for all targets you've configured.

At least, if it doesn't do that, then that's a bug!

@bors
Copy link
Contributor

bors commented Nov 3, 2016

☔ The latest upstream changes (presumably #37540) made this pull request unmergeable. Please resolve the merge conflicts.

This commit is a rewrite of the user-facing interface to the rustbuild build
system. The intention here is to make it much easier to compile/test the project
without having to remember weird rule names and such. An overall view of the new
interface is:

    # build everything
    ./x.py build

    # document everyting
    ./x.py doc

    # test everything
    ./x.py test

    # test libstd
    ./x.py test src/libstd

    # build libcore stage0
    ./x.py build src/libcore --stage 0

    # run stage1 run-pass tests
    ./x.py test src/test/run-pass --stage 1

The `src/bootstrap/bootstrap.py` script is now aliased as a top-level `x.py`
script. This `x` was chosen to be both short and easily tab-completable (no
collisions in that namespace!). The build system now accepts a "subcommand" of
what to do next, the main ones being build/doc/test.

Each subcommand then receives an optional list of arguments. These arguments are
paths in the source repo of what to work with. That is, if you want to test a
directory, you just pass that directory as an argument.

The purpose of this rewrite is to do away with all of the arcane renames like
"rpass" is the "run-pass" suite, "cfail" is the "compile-fail" suite, etc. By
simply working with directories and files it's much more intuitive of how to run
a test (just pass it as an argument).

The rustbuild step/dependency management was also rewritten along the way to
make this easy to work with and define, but that's largely just a refactoring of
what was there before.

The *intention* is that this support is extended for arbitrary files (e.g.
`src/test/run-pass/my-test-case.rs`), but that isn't quite implemented just yet.
Instead directories work for now but we can follow up with stricter path
filtering logic to plumb through all the arguments.
@alexcrichton
Copy link
Member Author

@bors: r=brson

@bors
Copy link
Contributor

bors commented Nov 3, 2016

📌 Commit a270b80 has been approved by brson

@bors
Copy link
Contributor

bors commented Nov 3, 2016

⌛ Testing commit a270b80 with merge f3c8663...

@bors
Copy link
Contributor

bors commented Nov 3, 2016

💔 Test failed - auto-mac-64-opt-rustbuild

@alexcrichton
Copy link
Member Author

@bors: retry

@bors
Copy link
Contributor

bors commented Nov 5, 2016

⌛ Testing commit a270b80 with merge 9374d2c...

@bors
Copy link
Contributor

bors commented Nov 5, 2016

💔 Test failed - auto-win-msvc-64-opt-rustbuild

bors added a commit that referenced this pull request Nov 5, 2016
bors added a commit that referenced this pull request Nov 6, 2016
@bors bors merged commit a270b80 into rust-lang:master Nov 6, 2016
@alexcrichton alexcrichton deleted the new-rustbuild branch November 6, 2016 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants