Ease debuging for the enduser #1
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
My apologies for the delay. Just merged. |
kohsuke
added a commit
that referenced
this pull request
May 27, 2014
While working on an elusive reference counting bug in remoting, I've found a bug in the reference counting scheme. Here's the sequence of events that would produce the problem: - French side exports object A as #1 - British side exports object B as #1 - British sends RemoteInvocationHandler to B in a serialized form to a French. - French receives & deserializes that into RemoteInvocationHandler(channel=British,oid=1) - French sends this RIH back to British. This can happen if RIH to B is a part of a reference graph of another Callable. Such RIH is called "going home" instance, as it's supposed to readResolve() into B. - British receives & deserializes it into RemoteInvocationHandler(channel=French,oid=1,goingHome=true) - ObjectInputStream on the British side immediately calls readResolve(), which returns B. - But British JVM still has RemoteInvocationHandler(channel=French,oid=1,goingHome=true), and when it later gets finalized, it ends up unexporting oid=1 exported from French, which is object A. The end result is that object A, which should have kept exported, gets unexported. So later attempt by a proxy to call back to object A will fail, because no such object is exported any more. It's unclear if this would solve the root cause of ZD-18447, but I think it's worth a try. Reference: ZD-18447
kohsuke
added a commit
that referenced
this pull request
Apr 15, 2015
I came across a hanging CLI client that was at the following point: "main" #1 prio=5 os_prio=0 tid=0x00c2b800 nid=0xf60 in Object.wait() [0x00bbf000 ] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:175) - locked <0x03656f68> (a [B) at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:146) at java.io.DataInputStream.readBoolean(Unknown Source) at hudson.cli.Connection.readBoolean(Connection.java:93) at hudson.cli.CLI.authenticate(CLI.java:560) at hudson.cli.CLI._main(CLI.java:471) at hudson.cli.CLI.main(CLI.java:382) The thread is stuck waiting for data to arrive from FastPipedOuputStream, which is exported and in turn waiting for the remote end to send more data. However, there's no Channel ReaderThread, and indeed the channel has already failed: Mar 08, 2015 11:10:50 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel CLI connection to https://cloudbees.ci.cloudbees.com/ java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.readShort(Unknown Source) at java.io.ObjectInputStream.readStreamHeader(Unknown Source) at java.io.ObjectInputStream.<init>(Unknown Source) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) The problem here is that the exported object (FastPipedOutputStream) doesn't notice that the actual writer (the other side of the channel) has disappeared, hence the reader blocks forever. Had FastPipedOutputStream got GC-ed, its finalizer had closed the stream and allowed the reader to receive an exception, but Channel was kept in the stack frame of the reader thread, so it is getting strongly referenced. In this commit, I'm fixing this problem twice. First, the change allows exported objects to notice a failure through ErrorPropagatingOutputStream, which FastPipedOutputStream and Pipe aleady implements. It seems reasonable for a pipe to be disrupted if the channel it's on is disrupted, and this would allow the "main" thread above to fail. Second, the change clears strong referenced to exported objects, allowing them to get GCed. FastPipedOutputStream would get GCed and it would have closed the writer end, resulting in the reader end getting EOF.
oleg-nenashev
added a commit
that referenced
this pull request
Nov 8, 2017
* [JENKINS-37566] - FindBugs: Unclosed stream in hudson.remoting.Capability * [JENKINS-37566] - FindBugs: Stream not closed on Exception path in InitializeJarCacheMain#copyFile * [JENKINS-37566] - Fix Util#makeResource and remove obsolete hack * Extra Util#makeResource() polishing * Deprecate obsolete Util#mkdirs() * Exceptional case in setLastModifiedTime * Handle exception case during temp file deletion in FileSystemJarCache * Synchronize ProxyWriter#closed in write() * Synchronization of ProxyWriter#channel * Get rid of the obsolete collection in Channel#properties, fix synchronization * Checksum#calculateFor() - no need to put if absent since there is a check above * UWF_UNWRITTEN_FIELD in ExportTable$Entry * NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE in JnlpAgentEndpointResolver. Now we use a more agressive check * UWF_FIELD_NOT_INITIALIZED_IN_CONSTRUCTOR in org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort * UPM_UNCALLED_PRIVATE_METHOD - hudson.remoting.ChunkedOutputStream.frameSize() * FindBugs: Revert the stream closure at emoting Capability, just a design to be reworked * Unrealistic NP_NULL_ON_SOME_PATH in NioChannelHub * EI_EXPOSE_REP in DiagnosedStreamCorruptionException (diagnostics code) * SE_TRANSIENT_FIELD_NOT_RESTORED in PreloadJarTask, RemoteClassLoader, RemoteInvocationHandler and UserRequest * Leftover false-positive UG_SYNC_SET_UNSYNC_GET in Channel * EI_EXPOSE_REP in ChannelBuilder#properties * Suppress DMI_NONSERIALIZABLE_OBJECT_WRITTEN in ClassLoaderHolder#writeObject() * Better handling of Streams in hudson.remoting.Capability (still weird) * Fix the Util#makeResource() behavior for nested folders * FileSystemJarCache: tmp file may be renamed * JENKINS-37566 - Modification of Channel#dumpDiagnostics() actually is not required after the merge of #122 * Disable DP_DO_INSIDE_DO_PRIVILEGED as per feeback from @kohsue in #118 * Fix the FindBugs filter * [NP_NULL_ON_SOME_PATH] - Prevent NPE in ExportTable#diagnoseInvalidObjectId() * [RV_RETURN_VALUE_IGNORED_BAD_PRACTICE] - Propagate exceptions when FileSystemJarCache#retrieve() fails to retrieve the target * [DP_CREATE_CLASSLOADER_INSIDE_DO_PRIVILEGED] - Ignore the warning as per discussion in #118 * [VO_VOLATILE_INCREMENT] - Suppress warning in Channel#send() * FindBugs - Which#jarFile() should not suppress all exceptions * FindBugs - Suppress URF_UNREAD_FIELD in Pipewindow#Real * [JENKINS-37566] - Channel#properties should be a ConcurrentHashMap Reason - prevent possible delays and deadlocks in the getter method. Addresses the feedback from @stephenc * [JENKINS-37566] - DiagnosedStreamCorruptionException readAhead/readBack handlers should use a more efficient clone() command * [JENKINS-37566] - Suppress UG_SYNC_SET_UNSYNC_GET after switching to the ConcurrentHashMap * [JENKINS-37566] - hudson.remoting.Util should be tolerant against InvalidPathException * [JENKINS-37566] - make ProxyWriter more asynchronous (follow-up to @stephenc’s review) * [JENKINS-37566] - Add a comment about race condition as suggested by @stephenc
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Endusers where reporting problems [1] to debug cases where a temp directory could not be created.
This pull adds some more information to the enduser in case the creation of the temp directory fails.
[1] https://issues.jenkins-ci.org/browse/JENKINS-11884