Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refresh roots #7782

Closed
wants to merge 60 commits into from
Closed

Refresh roots #7782

wants to merge 60 commits into from

Conversation

irengrig
Copy link
Contributor

No description provided.

irengrig and others added 26 commits March 20, 2019 17:53
Before, SymlinkTreeActions using SymlinkTreeStrategy hung in an indeterminate started-but-not-running state for their entire execution. Resolve this by posting a RunningActionEvent.

Closes #7731.

PiperOrigin-RevId: 239408774
The current implementation of these functions is very inefficient and
degrades overall performance significantly, especially when sandboxing is
enabled. However, that's almost the best we can do with a generic
algorithm.

To make room for optimizations that rely on specific file system features,
move these functions into the FileSystem class. I will supply a custom
implementation for UnixFileSystem later.

Note that this is intended to be a pure code move. I haven't applied any
improvements to the code nor tests yet (with the exception of cleaning up
docstrings).

Addresses #7527.

RELNOTES: None.
PiperOrigin-RevId: 239412965
…f NestedSets

that propagate an InterruptedException. This should have no performance impact
because the iterator for a NestedSet already calls .toCollection.

PiperOrigin-RevId: 239413057
This does not cover include scanning, only C++ compilation.

Progress on #6394.

PiperOrigin-RevId: 239436813
This means ensuring that all (non native) outputs of Starlark-defined transitions exist, are build settings, and are set to the correct type of value.

Wins along the way:
- properly propagate TransitionException through ConfiguredTargetFunction
- consolidate instances of COMMAND_LINE_OPTION_PREFIX

PiperOrigin-RevId: 239457521
FlagIdentifier.EXPERIMENTAL_ENABLE_ANDROID_MIGRATION_APIS flag.

RELNOTES: None.
PiperOrigin-RevId: 239482490
*** Reason for rollback ***

Breaks things inside Google. I'll roll it forward in a few days.

*** Original change description ***

Enable --incompatible_no_transitive_loads by default

Fixes #5636

Tested: https://buildkite.com/bazel/bazelisk-plus-incompatible-flags/builds/48

RELNOTES: --incompatible_no_transitive_loads is enabled by default.
PiperOrigin-RevId: 239485725
Add missing tests for the deleteTreesBelow entry point (expecting to leave
the given path untouched) and for corner cases like handling an unwritable
directory.

Addresses #7527.

RELNOTES: None.
PiperOrigin-RevId: 239489795
Before this cl, AndroidNdkRepositoryFunction generated CROSSTOOL file that configured the C++ toolchain for ndk. This cl changes this function to generate a Starlark rule providing CcToolchainConfigInfo, which is the "new" way of configuring C++ toolchains. This change should not be observable by ndk users.

As a result, generated ndk toolchain is forward compatible with incompatible changes:
* #7008
* #6861
* #7320
* #7075

RELNOTES: Android NDK C++ toolchain is now configured in Starlark. This should be a backwards compatible change, but in case of bugs blame unknown commit.
PiperOrigin-RevId: 239565213
If I/O is slow, FileWriteStrategy can take a while. Make sure we notify the UI that the action is at least running.

Closes #7787.

PiperOrigin-RevId: 239565386
…e_disable_expand_if_all_available_in_flag_set

Fixes #7008
Fixes #6861

This is an encore of c18be0b.

TESTED=https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/852#5c90f69e-c5d2-44c1-8067-0bde4cba6f6d (remote execution is fixed after syncing, verified locally)

RELNOTES: `--incompatible_disable_legacy_crosstool_fields` has been flipped (#6861)
`--incompatible_disable_expand_if_all_available_in_flag_set` has been flipped (#7008)
PiperOrigin-RevId: 239567807
Issue #5380

RELNOTES: None.
PiperOrigin-RevId: 239568494
…rget label to make it clearer.

RELNOTES: None
PiperOrigin-RevId: 239569522
Make the literals passed to (EXPECT|ASSERT)_EQ unsigned when required. This suppresses a lot of GCC warnings.

Closes #7788.

PiperOrigin-RevId: 239580168
Issue #5380

RELNOTES: None.
PiperOrigin-RevId: 239580767
Partial commit for third_party/*, see #7785.

Signed-off-by: iirina <[email protected]>
The flag bep_publish_used_heap_size_post_build is not meant to be
exposed to other commands than build since it has no effect there.

RELNOTES: None

Closes #7790.

PiperOrigin-RevId: 239586196
Make deleteTreesBelow faster by assuming that the directories to be deleted
are readable and writable.  We create most of the trees we delete via this
function anyway, so we know that they are accessible for the most part.

The previous code was blindly resetting read/write/execute permissions for
each traversed directory, and was doing so individually, which means we
issued 3 extra syscalls per directory.

And on Unix file systems, go even further by taking advantage of the fact
that readdir returns the type of each entry: there is no need to issue a
separate stat for each entry to determine if it is a subdirectory or not.
Do this from our JNI code because there really is no reason to pay the
cost of going in an out of Java for each file: we are traversing very
large directory trees, so every bit helps.

A fully-local build of a large iOS app on a Mac Pro 2013 shows that this
reduces build times from about 7300s to 5100s.  A build of a similar app
on a MacBook Pro 2015 shows a reduction from 7500s to 5400s.  The impact
on these builds using dynamic execution is much smaller, and there is
no observable improvement in smaller builds.

Addresses #7527.

RELNOTES: None.
PiperOrigin-RevId: 239594433
`third_party/jarjar` was moved out of the `@bazel_tools` when the Java tools were moved to a remote repository. Adding the package back because there are users referencing it directly.

Fixes #7767.

Closes #7785.

PiperOrigin-RevId: 239595898
buchgr and others added 22 commits March 25, 2019 13:27
split out from #7778

Closes #7791.

PiperOrigin-RevId: 239638546
…ternal and remove references to gmaven_rules

Closes #7789.

PiperOrigin-RevId: 239653154
RELNOTES: None
PiperOrigin-RevId: 239657747
…lect not always being split transitions.

RELNOTES: None.
PiperOrigin-RevId: 239661406
URL.openConnection can throw an IllegalArgumentException, e.g., in the
JceSecurityManager. While we cannot fix these exceptions, we can at least
report them in a more user-friendly way than an internal stack trace. So
catch and throw an exception that is reported as a permanent error attempting
to fetch the given URL.

Issue #6952.

Change-Id: I2bb5d12535d54046e2cf342acabcb28c2066eed7
PiperOrigin-RevId: 239762567
Add the needed load statements to the example WORKSPACE files. In
this way, the examples work if blindly copied and pasted.

Change-Id: I661ea632781f575f51f71d6e53a405cf81c2de2e
PiperOrigin-RevId: 239764071
The console output stream is turned on by default for quite a while
now, and the decision is made that it should stay the default, and
in the long run the only, console output provided by Bazel. Rename
the respective flags accordingly to avoid giving a wrong impression
about its state.

Change-Id: Iad6240c4f615aae64fc721cd2e26fa5ee30b000d
PiperOrigin-RevId: 239781052
*** Reason for rollback ***

Roll forward

*** Original change description ***

Automated rollback of commit f7f5415.

*** Reason for rollback ***

Breaks things inside Google. I'll roll it forward in a few days.

*** Original change description ***

Enable --incompatible_no_transitive_loads by default

Fixes #5636

Tested: https://buildkite.com/bazel/bazelisk-plus-incompatible-flags/builds/48

RELNOTES: --incompatible_no_transitive_loads is enabled by default.
PiperOrigin-RevId: 239798602
Part of #7814.

Closes #7815.

PiperOrigin-RevId: 239811685
Closes #7804.

PiperOrigin-RevId: 239819422
EvaluationContext has 2 members for different configurations it is run with
(either is set, never both):
  1. Integer numThreads -- this allows delegating the decision of what kind
  ExecutorService to create to the user
  2. Supplier<ExecutorService> executorService -- factory of ExecutorService
  objects -- makes the user tied to what it returns

Some places in the code are interested in knowing the parallelism of the
executor, which currently is achieved by assuming the kind of thread pool we
would use in case numThreads is not set.

Change the EvaluationContext to always require and store the parallelism (even
if the ExecutorService supplier is specified). Rename the getter to
'getParallelism' to avoid confusion with the old concept.

PiperOrigin-RevId: 239826228
…ge is unexpectedly absent.

PiperOrigin-RevId: 239837427
Currently, both android_test and android_instrumentation_test explicitly filter out all R classes from the test's deploy.jar when a test instruments another app rather than filtering out only the duplicate R classes.

Unfortunately, the android_test and android_instrumentation_test use differing CheckHashMismatchMode filter settings WARN and NONE respectively. Where NONE will cause the filterer to remove all duplicate classes from the test's deploy jar and WARN will not filter duplicate classes when the hash values differ

As such this cl introduces two flags:
  1. Switch the CheckHashMismatchMode to NONE for android_test while keeping intact the whitelist for making hash mismatches a failure
  2. Disable the explicit filtering of R classes.

RELNOTES: None.
PiperOrigin-RevId: 239852282
Fixes #5815

Tested: https://buildkite.com/bazel/bazelisk-plus-incompatible-flags/builds/49

RELNOTES: --incompatible_bzl_disallow_load_after_statement is enabled
PiperOrigin-RevId: 239854652
There will typically only be a single instance of PlatformMappingValue in a single command but it will be used for several configurations. Avoid re-parsing the mappings for each configuration by transiently caching the parsing results.

RELNOTES: None
PiperOrigin-RevId: 239862008
Note that we will ignore any interrupted actions for the purposes of
computing the critical path of the build, since there were no completion
events for those actions.

Progress on #6394.

PiperOrigin-RevId: 240095667
…int to CcToolchainConfigInfo reference instead of CROSSTOOL reference.

RELNOTES: None
PiperOrigin-RevId: 240096872
This rewrites execute() in terms of beginExecution and the resulting
continuation(s), and removes the old implementation. It implicitly
switches all the existing tests to the new code paths.

There's a small risk that this introduces a subtle behavioral
difference that isn't caught by the existing tests, but I'd say we
have pretty good test coverage.

Progress on #6394.

PiperOrigin-RevId: 240097352
In order to migrate C++ rules to platforms, we need the access to the C++ configuration fragment from the caller rule in Starlark APIs. All existing APIs have already access to it, but cc_common.configure_features doesn't. This incompatible change adds a ctx argument to configure_features.

#7793
#6516

RELNOTES: Added `--incompatible_require_ctx_in_configure_features`, see #7793 for details.
PiperOrigin-RevId: 240099411
@googlebot
Copy link

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.