HBASE-23333 Provide call context around timeouts and other failure scenarios #872

ndimiduk · 2019-11-23T00:13:04Z

No description provided.

ndimiduk · 2019-11-23T00:14:00Z

Initial commit simply include the output of Call#toString where before we only have call.id. Looking to see if there's a sensible way to include a call stack that tracks up into the application code. Will be back.

Apache-HBase · 2019-11-23T01:04:25Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 33s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-0 ⚠️	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	5m 25s	master passed
+1 💚	compile	0m 26s	master passed
+1 💚	checkstyle	0m 32s	master passed
+1 💚	shadedjars	4m 34s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
+0 🆗	spotbugs	1m 7s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 6s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 50s	the patch passed
+1 💚	compile	0m 25s	the patch passed
+1 💚	javac	0m 25s	the patch passed
+1 💚	checkstyle	0m 32s	hbase-client: The patch generated 0 new + 7 unchanged - 3 fixed = 7 total (was 10)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedjars	4m 33s	patch has no errors when building our shaded downstream artifacts.
+1 💚	hadoopcheck	15m 45s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 💚	javadoc	0m 23s	the patch passed
+1 💚	findbugs	1m 12s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 53s	hbase-client in the patch passed.
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		49m 8s

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/1/artifact/out/Dockerfile
GITHUB PR	#872
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux 435e5cb8da54 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-872/out/precommit/personality/provided.sh
git revision	master / `ee730c8`
Default Java	1.8.0_181
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/1/testReport/
Max. process+thread count	294 (vs. ulimit of 10000)
modules	C: hbase-client U: hbase-client
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/1/console
versions	git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

bharathv

nice cleanup. Couple of optional nits.

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java

bharathv · 2019-11-27T17:56:47Z

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/Call.java

@@ -78,8 +78,13 @@ protected Call(int id, final Descriptors.MethodDescriptor md, Message param,

  @Override
  public String toString() {
-    return "callId: " + this.id + " methodName: " + this.md.getName() + " param {"
-        + (this.param != null ? ProtobufUtil.getShortTextFormat(this.param) : "") + "}";
+    return new ToStringBuilder(this, ToStringStyle.SHORT_PREFIX_STYLE)


ndimiduk · 2019-12-04T23:20:50Z

I found more-or-less what I was looking for over on HBASE-22316. Somehow the piece of stack trace i found in my logs was omitted. Settling for this for now.

Apache-HBase · 2019-12-05T00:20:06Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 55s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ master Compile Tests _
+0 🆗	mvndep	0m 46s	Maven dependency ordering for branch
+1 💚	mvninstall	5m 15s	master passed
+1 💚	compile	0m 49s	master passed
+1 💚	checkstyle	0m 59s	master passed
+1 💚	shadedjars	4m 35s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 45s	master passed
+0 🆗	spotbugs	1m 4s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 49s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 15s	Maven dependency ordering for patch
+1 💚	mvninstall	4m 56s	the patch passed
+1 💚	compile	0m 48s	the patch passed
+1 💚	javac	0m 48s	the patch passed
+1 💚	checkstyle	0m 26s	The patch passed checkstyle in hbase-common
+1 💚	checkstyle	0m 31s	hbase-client: The patch generated 0 new + 7 unchanged - 3 fixed = 7 total (was 10)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedjars	4m 35s	patch has no errors when building our shaded downstream artifacts.
+1 💚	hadoopcheck	15m 38s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 💚	javadoc	0m 45s	the patch passed
+1 💚	findbugs	2m 15s	the patch passed
		_ Other Tests _
+1 💚	unit	3m 15s	hbase-common in the patch passed.
+1 💚	unit	1m 55s	hbase-client in the patch passed.
+1 💚	asflicense	0m 26s	The patch does not generate ASF License warnings.
		59m 42s

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/2/artifact/out/Dockerfile
GITHUB PR	#872
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux aa33225b15ec 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-872/out/precommit/personality/provided.sh
git revision	master / `5e34a59`
Default Java	1.8.0_181
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/2/testReport/
Max. process+thread count	292 (vs. ulimit of 10000)
modules	C: hbase-common hbase-client U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/2/console
versions	git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache9

What is the new error message look like? Will it be too long as we log the whole call?

Apache9 · 2019-12-05T03:19:14Z

hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestFutureUtils.java

@@ -80,6 +80,8 @@ public void testRecordStackTrace() throws IOException {
        startsWith("org.apache.hadoop.hbase.util.TestFutureUtils.testRecordStackTrace"));
      assertTrue(Stream.of(elements)
        .anyMatch(element -> element.toString().contains("--------Future.get--------")));
+    } catch (Throwable t) {
+      throw new AssertionError("Caught unexpected Throwable", t);


What's this...

This is to ensure that the test fails when the exception we expect isn't thrown. It would be simply fail() except that api doesn't let me retain the context provided by t.

ndimiduk · 2019-12-05T23:17:15Z

Maybe this is too verbose?

org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=11, exceptions:
Thu Dec 05 23:11:14 UTC 2019, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60160: Call to redacted:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=4,methodName=Scan,param=region { type: REGION_NAME value: "hbase:meta,,1" } scan { column { family: "info" } start_row: “my_table,,” stop_row: “my_table ,," time_range { from: 0 to: 9223372036854775807 } max_versions: 1 cache_blocks: true max_result_size: 2097152 caching: 100 include_stop_row: false readType: PREAD } number_of_rows: 100 close_scanner: false client_handles_partials: true client_handles_heartbeats: true track_scan_metrics: false limit_of_rows: 2147483647], waitTime=60021, rpcTimeout=59999 row ‘my_table,,’ on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=redacted,16020,1575517640564, seqNum=-1

ndimiduk · 2019-12-06T00:41:31Z

Rebased onto master and reduced the verbosity to just call.id and call.methodName. It would be nice to selectively include parameters, such as all non-default values. Could update the log sites to log less when the redundant values are already in the params...

What do you folks think?

Apache-HBase · 2019-12-06T01:43:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ master Compile Tests _
+0 🆗	mvndep	0m 35s	Maven dependency ordering for branch
+1 💚	mvninstall	5m 16s	master passed
+1 💚	compile	0m 50s	master passed
+1 💚	checkstyle	1m 0s	master passed
+1 💚	shadedjars	4m 36s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 45s	master passed
+0 🆗	spotbugs	1m 7s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 53s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 15s	Maven dependency ordering for patch
+1 💚	mvninstall	4m 56s	the patch passed
+1 💚	compile	0m 50s	the patch passed
+1 💚	javac	0m 50s	the patch passed
+1 💚	checkstyle	0m 25s	The patch passed checkstyle in hbase-common
+1 💚	checkstyle	0m 31s	hbase-client: The patch generated 0 new + 7 unchanged - 3 fixed = 7 total (was 10)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedjars	4m 34s	patch has no errors when building our shaded downstream artifacts.
+1 💚	hadoopcheck	15m 49s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 💚	javadoc	0m 44s	the patch passed
+1 💚	findbugs	2m 9s	the patch passed
		_ Other Tests _
+1 💚	unit	3m 9s	hbase-common in the patch passed.
+1 💚	unit	1m 52s	hbase-client in the patch passed.
+1 💚	asflicense	0m 26s	The patch does not generate ASF License warnings.
		57m 51s

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/3/artifact/out/Dockerfile
GITHUB PR	#872
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux 5620002d1c93 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-872/out/precommit/personality/provided.sh
git revision	master / `6d251ef`
Default Java	1.8.0_181
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/3/testReport/
Max. process+thread count	292 (vs. ulimit of 10000)
modules	C: hbase-common hbase-client U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/3/console
versions	git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

ndimiduk · 2019-12-09T16:38:50Z

Still have objections @Apache9 ?

bharathv

Patch seems fine to me, fwiw.

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/Call.java

Apache-HBase · 2020-01-03T18:48:46Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 34s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ master Compile Tests _
+0 🆗	mvndep	0m 35s	Maven dependency ordering for branch
+1 💚	mvninstall	5m 21s	master passed
+1 💚	compile	0m 49s	master passed
+1 💚	checkstyle	0m 58s	master passed
+1 💚	shadedjars	4m 37s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 46s	master passed
+0 🆗	spotbugs	1m 5s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 50s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 15s	Maven dependency ordering for patch
+1 💚	mvninstall	4m 54s	the patch passed
+1 💚	compile	0m 49s	the patch passed
+1 💚	javac	0m 49s	the patch passed
+1 💚	checkstyle	0m 26s	The patch passed checkstyle in hbase-common
+1 💚	checkstyle	0m 31s	hbase-client: The patch generated 0 new + 7 unchanged - 3 fixed = 7 total (was 10)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedjars	4m 31s	patch has no errors when building our shaded downstream artifacts.
+1 💚	hadoopcheck	15m 38s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 💚	javadoc	0m 44s	the patch passed
+1 💚	findbugs	2m 15s	the patch passed
		_ Other Tests _
+1 💚	unit	3m 15s	hbase-common in the patch passed.
+1 💚	unit	1m 58s	hbase-client in the patch passed.
+1 💚	asflicense	0m 24s	The patch does not generate ASF License warnings.
		58m 7s

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/4/artifact/out/Dockerfile
GITHUB PR	#872
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux 79f2b61f1698 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-872/out/precommit/personality/provided.sh
git revision	master / `85ecdd1`
Default Java	1.8.0_181
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/4/testReport/
Max. process+thread count	286 (vs. ulimit of 10000)
modules	C: hbase-common hbase-client U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/4/console
versions	git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

bharathv approved these changes Nov 27, 2019

View reviewed changes

virajjasani approved these changes Nov 29, 2019

View reviewed changes

ndimiduk force-pushed the 23333-call-context branch 2 times, most recently from a56df5f to c7953dd Compare December 4, 2019 23:18

Apache9 requested changes Dec 5, 2019

View reviewed changes

ndimiduk force-pushed the 23333-call-context branch from c7953dd to 44ef309 Compare December 6, 2019 00:39

bharathv approved these changes Dec 11, 2019

View reviewed changes

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/Call.java Outdated Show resolved Hide resolved

HBASE-23333 Include Call.toShortString() in sendCall exceptions

d2ffc92

ndimiduk force-pushed the 23333-call-context branch from 44ef309 to d2ffc92 Compare January 3, 2020 17:45

ndimiduk merged commit af3bb1f into apache:master Jan 3, 2020

ndimiduk deleted the 23333-call-context branch January 3, 2020 19:09

This was referenced Jan 3, 2020

HBASE-23333 Include Call.toShortString() in sendCall exceptions #984

Merged

HBASE-23333 Include Call.toShortString() in sendCall exceptions #985

Merged

HBASE-23333 Include Call.toShortString() in sendCall exceptions #986

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-23333 Provide call context around timeouts and other failure scenarios #872

HBASE-23333 Provide call context around timeouts and other failure scenarios #872

ndimiduk commented Nov 23, 2019

ndimiduk commented Nov 23, 2019

Apache-HBase commented Nov 23, 2019

bharathv left a comment

bharathv Nov 27, 2019

ndimiduk commented Dec 4, 2019

Apache-HBase commented Dec 5, 2019

Apache9 left a comment

Apache9 Dec 5, 2019

ndimiduk Dec 5, 2019

ndimiduk commented Dec 5, 2019

ndimiduk commented Dec 6, 2019

Apache-HBase commented Dec 6, 2019

ndimiduk commented Dec 9, 2019

bharathv left a comment

Apache-HBase commented Jan 3, 2020

HBASE-23333 Provide call context around timeouts and other failure scenarios #872

HBASE-23333 Provide call context around timeouts and other failure scenarios #872

Conversation

ndimiduk commented Nov 23, 2019

ndimiduk commented Nov 23, 2019

Apache-HBase commented Nov 23, 2019

bharathv left a comment

Choose a reason for hiding this comment

bharathv Nov 27, 2019

Choose a reason for hiding this comment

ndimiduk commented Dec 4, 2019

Apache-HBase commented Dec 5, 2019

Apache9 left a comment

Choose a reason for hiding this comment

Apache9 Dec 5, 2019

Choose a reason for hiding this comment

ndimiduk Dec 5, 2019

Choose a reason for hiding this comment

ndimiduk commented Dec 5, 2019

ndimiduk commented Dec 6, 2019

Apache-HBase commented Dec 6, 2019

ndimiduk commented Dec 9, 2019

bharathv left a comment

Choose a reason for hiding this comment

Apache-HBase commented Jan 3, 2020