Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NPE in CompletionFunction #22741

Closed
fmeum opened this issue Jun 13, 2024 · 3 comments
Closed

NPE in CompletionFunction #22741

fmeum opened this issue Jun 13, 2024 · 3 comments
Assignees
@coeuvre
Labels
P1 I'll work on this now. (Assignee required) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug

Comments

@fmeum
Copy link
Collaborator

fmeum commented Jun 13, 2024

Description of the bug:

A failure during remote execution resulted in this server crash:

java.lang.RuntimeException: Unrecoverable error while evaluating node 'TargetCompletionKey{topLevelArtifactContext=com.google.devtools.build.lib.analysis.TopLevelArtifactContext@90904c3b, actionLookupKey=ConfiguredTargetKey{label=//enterprise/server/test/integration/remote_cache:remote_cache_test, config=BuildConfigurationKey[516e39aa8c2308bf34cdfc86e47e7aa61251bb785760a90a2ab72446f66a2d73]}, willTest=true}' (requested by nodes 'TestCompletionKey{configuredTargetKey=ConfiguredTargetKey{label=//enterprise/server/test/integration/remote_cache:remote_cache_test, config=BuildConfigurationKey[516e39aa8c2308bf34cdfc86e47e7aa61251bb785760a90a2ab72446f66a2d73]}, topLevelArtifactContext=com.google.devtools.build.lib.analysis.TopLevelArtifactContext@90904c3b, exclusiveTesting=false}')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:550)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException: FailureDetail missing category submessage: message: "io.grpc.StatusRuntimeException: CANCELLED: Failed to read message."

	at com.google.devtools.build.lib.skyframe.CompletionFunction.ensureToplevelArtifacts(CompletionFunction.java:449)
	at com.google.devtools.build.lib.skyframe.CompletionFunction.compute(CompletionFunction.java:329)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
	... 7 more
Caused by: java.lang.NullPointerException: FailureDetail missing category submessage: message: "io.grpc.StatusRuntimeException: CANCELLED: Failed to read message."

	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:1010)
	at com.google.devtools.build.lib.util.DetailedExitCode.getCategorySubmessage(DetailedExitCode.java:147)
	at com.google.devtools.build.lib.util.DetailedExitCode.getNumericExitCode(DetailedExitCode.java:116)
	at com.google.devtools.build.lib.util.DetailedExitCode.getExitCode(DetailedExitCode.java:48)
	at com.google.devtools.build.lib.util.DetailedExitCode.of(DetailedExitCode.java:87)
	at com.google.devtools.build.lib.skyframe.CompletionFunction.lambda$ensureToplevelArtifacts$1(CompletionFunction.java:435)
	at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:205)
	at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:192)
	at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:134)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
	at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:105)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
	at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:850)
	at com.google.common.util.concurrent.CombinedFuture$AsyncCallableInterruptibleTask.setValue(CombinedFuture.java:175)
	at com.google.common.util.concurrent.CombinedFuture$AsyncCallableInterruptibleTask.setValue(CombinedFuture.java:153)
	at com.google.common.util.concurrent.CombinedFuture$CombinedFutureInterruptibleTask.afterRanInterruptiblySuccess(CombinedFuture.java:129)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:88)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.CombinedFuture$CombinedFutureInterruptibleTask.execute(CombinedFuture.java:108)
	at com.google.common.util.concurrent.CombinedFuture.handleAllCompleted(CombinedFuture.java:65)
	at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:301)
	at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:283)
	at com.google.common.util.concurrent.AggregateFuture.lambda$init$1(AggregateFuture.java:181)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
	at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:55)
	at com.google.devtools.build.lib.remote.util.RxFutures$1.onError(RxFutures.java:221)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableFromSingle$CompletableFromSingleObserver.onError(CompletableFromSingle.java:41)
	at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.tryOnError(SingleCreate.java:95)
	at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onError(SingleCreate.java:81)
	at com.google.devtools.build.lib.remote.util.AsyncTaskCache$1.onError(AsyncTaskCache.java:339)
	at com.google.devtools.build.lib.remote.util.AsyncTaskCache$Execution.onError(AsyncTaskCache.java:205)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableToSingle$ToSingle.onError(CompletableToSingle.java:73)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableUsing$UsingObserver.onError(CompletableUsing.java:165)
	at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek$CompletableObserverImplementation.onError(CompletablePeek.java:95)
	at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek$CompletableObserverImplementation.onError(CompletablePeek.java:95)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate$Emitter.tryOnError(CompletableCreate.java:91)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate$Emitter.onError(CompletableCreate.java:77)
	at com.google.devtools.build.lib.remote.util.RxFutures$OnceCompletableOnSubscribe$1.onFailure(RxFutures.java:102)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1119)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
	at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:55)
	at com.google.devtools.build.lib.remote.RemoteCache$3.onFailure(RemoteCache.java:381)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1119)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
	at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:850)
	at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:216)
	at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:192)
	at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:144)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
	at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:105)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
	at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:850)
	at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:216)
	at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:192)
	at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:144)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
	at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:850)
	at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:216)
	at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:192)
	at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:144)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
	at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:105)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
	at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:55)
	at com.google.devtools.build.lib.remote.util.RxFutures$2.onError(RxFutures.java:259)
	at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onError(SingleFlatMap.java:117)
	at io.reactivex.rxjava3.internal.operators.single.SingleUsing$UsingSingleObserver.onError(SingleUsing.java:180)
	at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.tryOnError(SingleCreate.java:95)
	at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onError(SingleCreate.java:81)
	at com.google.devtools.build.lib.remote.util.RxFutures$OnceSingleOnSubscribe$1.onFailure(RxFutures.java:172)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1119)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
	at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:55)
	at com.google.devtools.build.lib.remote.GrpcCacheClient$1.onError(GrpcCacheClient.java:455)
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.google.devtools.build.lib.remote.NetworkTimeInterceptor$NetworkTimeCall$1.onClose(NetworkTimeInterceptor.java:81)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

Which category does this issue belong to?

Remote Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

No response

What is the output of bazel info release?

No response

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response
@github-actions github-actions bot added the team-Remote-Exec Issues and PRs for the Execution (Remote) team label Jun 13, 2024
@coeuvre coeuvre self-assigned this Jun 17, 2024
@coeuvre coeuvre added P1 I'll work on this now. (Assignee required) and removed untriaged labels Jun 17, 2024
@coeuvre
Copy link
Member

coeuvre commented Jun 17, 2024

@bazel-io fork 7.2.1

@bazel-io bazel-io mentioned this issue Jun 17, 2024
Closed
@coeuvre
Copy link
Member

coeuvre commented Jun 17, 2024

@bazel-io fork 7.3.0

@bazel-io bazel-io mentioned this issue Jun 17, 2024
Closed
Wyverald pushed a commit that referenced this issue Jun 20, 2024
@coeuvre @Wyverald
Add missing category submessage.
4910d58 
Fixes #22741.

PiperOrigin-RevId: 643932127
Change-Id: Ia9566acb7b828224679ad57b71644281a8e52db4
Wyverald pushed a commit that referenced this issue Jun 20, 2024
@coeuvre @Wyverald
Add missing category submessage.
e87afc2 
Fixes #22741.

PiperOrigin-RevId: 643932127
Change-Id: Ia9566acb7b828224679ad57b71644281a8e52db4
This was referenced Jun 20, 2024
Merged
Merged
github-merge-queue bot pushed a commit that referenced this issue Jun 20, 2024
@Wyverald @coeuvre
[7.2.1] Add missing category submessage. (#22828)
3278ffd 
Fixes #22741.

PiperOrigin-RevId: 643932127
Change-Id: Ia9566acb7b828224679ad57b71644281a8e52db4

Co-authored-by: Googler <[email protected]>
github-merge-queue bot pushed a commit that referenced this issue Jun 21, 2024
@Wyverald @coeuvre @meteorcloudy
[7.3.0] Add missing category submessage. (#22827)
5562e87 
Fixes #22741.

PiperOrigin-RevId: 643932127
Change-Id: Ia9566acb7b828224679ad57b71644281a8e52db4

Co-authored-by: Googler <[email protected]>
Co-authored-by: Yun Peng <[email protected]>
@iancha1992
Copy link
Member

A fix for this issue has been included in Bazel 7.2.1 RC2. Please test out the release candidate and report any issues as soon as possible.
If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=7.2.1rc2. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@coeuvre coeuvre

Labels
P1 I'll work on this now. (Assignee required) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@fmeum @coeuvre @sgowroji @iancha1992 @satyanandak