-
Notifications
You must be signed in to change notification settings - Fork 4.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Async action execution is implemented as layered, nested state machines, where each state machine corresponds to a specific behavior. Each instance of a continuation is a state in a state machine, and some states run entire state machines at the next level. There are now 4 layers of continuations: 1. The ActionExecutionFunction uses a continuation for each input-discovering action to reduce the overhead of function restarts. 2. The SkyframeActionExecutor uses a continuation for each group of shared actions (as determined by the name of the primary output). Note in particular how this differs in scope from the first layer. 3. Action provides a generic level of continuations for actions to use to represent any sequence of internal work needed for that action. SpawnAction specifically uses this to represent the additional steps wrapping the lower-level spawn execution. 4. The SpawnContinuation represents the execution of a single spawn. This includes fallback from remote to local execution, as well as the work necessary to implement tree artifacts. Tree artifacts may be implemented by wrapping the given spawn in a shell script that unpacks any tree artifacts present in its inputs, as well as a subsequent spawn that unpacks any tree artifacts present in its outputs (post spawn execution). We could consider merging some of these layers. However, it is not trivial to extend or modify an existing state machine. For example, it is not straightforward to merge 3 and 4. Layer 4 represents the state machine required by the SpawnActionContext, which may differ based on the locality (remote or local) or sandboxing behavior of that strategy, whereas 3 represents the state machine required by the action. Merging them would require that actions are aware of all possible state machines required for any SpawnActionContext, or alternatively pass some information through to the SpawnActionContext (a continuation?) that is then transitioned to after the strategy-specific state machine is completed. While this may save us some memory, it may also end up increasing complexity. Progress on #6394. PiperOrigin-RevId: 236597242
- Loading branch information
1 parent
00ac93a
commit 40ce64e
Showing
4 changed files
with
161 additions
and
40 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
100 changes: 100 additions & 0 deletions
100
src/main/java/com/google/devtools/build/lib/actions/SpawnContinuation.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| // Copyright 2019 The Bazel Authors. All rights reserved. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
| package com.google.devtools.build.lib.actions; | ||
|
|
||
| import com.google.common.collect.ImmutableList; | ||
| import com.google.common.util.concurrent.ListenableFuture; | ||
| import java.util.List; | ||
|
|
||
| /** | ||
| * A representation of a (potentially) multi-step spawn execution, which can return multiple | ||
| * results. This covers cases like remote/local fallback as well as tree artifact packing/unpacking, | ||
| * which both require multiple attempts at running a spawn, and therefore can have multiple results. | ||
| * | ||
| * <p>This is intentionally similar to {@link ActionContinuationOrResult}, which will often wrap one | ||
| * of these. | ||
| * | ||
| * <p>Any client of this class <b>must</b> first call {@link #isDone} before calling any of the | ||
| * other methods. If {@link #isDone} returns true, then {@link #getFuture} and {@link #execute} must | ||
| * throw {@link IllegalStateException}, but {@link #get} must return a valid value. Use {@link | ||
| * #immediate} to construct such an instance. | ||
| * | ||
| * <p>Otherwise, {@link #getFuture} must return a non-null value, and {@link #execute} must not | ||
| * throw {@link IllegalStateException}, whereas {@link #get} must throw {@link | ||
| * IllegalStateException}. | ||
| */ | ||
| public abstract class SpawnContinuation { | ||
| public static SpawnContinuation immediate(SpawnResult... spawnResults) { | ||
| return new Finished(ImmutableList.copyOf(spawnResults)); | ||
| } | ||
|
|
||
| public static SpawnContinuation immediate(List<SpawnResult> spawnResults) { | ||
| return new Finished(ImmutableList.copyOf(spawnResults)); | ||
| } | ||
|
|
||
| /** | ||
| * Runs the state machine represented by the given continuation to completion, blocking as | ||
| * necessary until all asynchronous computations finish, and the final continuation is done. Then | ||
| * returns the list of spawn results (calling {@link SpawnContinuation#get}). | ||
| * | ||
| * <p>This method provides backwards compatibility for the cases where a method that's defined as | ||
| * blocking obtains a continuation and needs the result before it can return. Over time, this | ||
| * method should become less common as more actions are rewritten to support async execution. | ||
| */ | ||
| public static List<SpawnResult> completeBlocking(SpawnContinuation continuation) | ||
| throws ExecException, InterruptedException { | ||
| while (!continuation.isDone()) { | ||
| continuation = continuation.execute(); | ||
| } | ||
| return continuation.get(); | ||
| } | ||
|
|
||
| public boolean isDone() { | ||
| return false; | ||
| } | ||
|
|
||
| public abstract ListenableFuture<?> getFuture(); | ||
|
|
||
| public abstract SpawnContinuation execute() throws ExecException, InterruptedException; | ||
|
|
||
| public List<SpawnResult> get() { | ||
| throw new IllegalStateException(); | ||
| } | ||
|
|
||
| private static final class Finished extends SpawnContinuation { | ||
| private final List<SpawnResult> spawnResults; | ||
|
|
||
| Finished(List<SpawnResult> spawnResults) { | ||
| this.spawnResults = spawnResults; | ||
| } | ||
|
|
||
| public boolean isDone() { | ||
| return true; | ||
| } | ||
|
|
||
| @Override | ||
| public ListenableFuture<?> getFuture() { | ||
| throw new IllegalStateException(); | ||
| } | ||
|
|
||
| @Override | ||
| public SpawnContinuation execute() { | ||
| throw new IllegalStateException(); | ||
| } | ||
|
|
||
| public List<SpawnResult> get() { | ||
| return spawnResults; | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters