HBASE-25596: Fix NPE and avoid permanent unreplicated data due to EOF #2975

sandeepvinayak · 2021-02-22T21:52:25Z

Please find the details in jira:
https://issues.apache.org/jira/browse/HBASE-25596

sandeepvinayak · 2021-02-22T22:01:51Z

.../java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceWALReaderThread.java

+          // After we removed the WAL from the queue, we should
+          // try shipping the existing batch of entries, we do not want to reset
+          // stream since entry stream doesn't have the correct data at this point
+          isBatchQueuedToBeShipped(entryStream, batch, hasMoreData, true);


This is the fix to avoid permanent data loss.

...server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java

apurtell · 2021-02-22T22:04:08Z

...server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java

@@ -840,7 +844,7 @@ public void uncaughtException(final Thread t, final Throwable e) {

    // If this is a recovered queue, the queue is already full and the first log
    // normally has a position (unless the RS failed between 2 logs)
-    private long getRecoveredQueueStartPos(long startPosition) {
+    public long getRecoveredQueueStartPos(long startPosition) {


Is this change needed? I don't see any changes to a caller of this method in this patch

No, I will revert it.

.../java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceWALReaderThread.java

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java

apurtell · 2021-02-22T22:17:50Z

...erver/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntryStream.java

+    assertEquals(0, queue.size());
+  }
+
+  @Test


hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java

xcangCRM · 2021-02-23T00:02:09Z

...erver/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntryStream.java

+        queue, 0, fs, conf, getDummyFilter(),
+        new MetricsSource("1"), (ReplicationSource) source);
+    reader.run();
+


do you need to do some wait here? How did you drain the queue? I might be missing something here. thanks

@xcangCRM i think you are confusing it with .start() method which is async call. We are executing .run() which is sync call.

.../java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceWALReaderThread.java

xcangCRM · 2021-02-23T00:08:11Z

.../java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceWALReaderThread.java

+   * @throws InterruptedException throws interrupted exception
+   * @throws IOException throws io exception from stream
+   */
+  private boolean updateBatchAndAddInShippingQueue(WALEntryStream entryStream, WALEntryBatch batch,


nit:
name is bit long,
how about updateBatchAndShippingQueue

.../java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceWALReaderThread.java

Apache-HBase · 2021-02-23T01:24:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 52s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 3 new or modified test files.
		_ branch-1 Compile Tests _
+1 💚	mvninstall	11m 17s	branch-1 passed
+1 💚	compile	0m 59s	branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚	compile	1m 0s	branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚	checkstyle	2m 19s	branch-1 passed
+1 💚	shadedjars	4m 33s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 7s	branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚	javadoc	1m 0s	branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+0 🆗	spotbugs	4m 13s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	4m 8s	branch-1 passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 51s	the patch passed
+1 💚	compile	1m 3s	the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚	javac	1m 3s	the patch passed
+1 💚	compile	1m 4s	the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚	javac	1m 4s	the patch passed
+1 💚	checkstyle	2m 14s	hbase-server: The patch generated 0 new + 0 unchanged - 10 fixed = 0 total (was 10)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedjars	4m 40s	patch has no errors when building our shaded downstream artifacts.
+1 💚	hadoopcheck	6m 48s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2.
+1 💚	javadoc	0m 57s	hbase-server-jdkAzulSystems,Inc.-1.8.0_262-b19 with JDK Azul Systems, Inc.-1.8.0_262-b19 generated 0 new + 6 unchanged - 2 fixed = 6 total (was 8)
+1 💚	javadoc	1m 3s	hbase-server-jdkAzulSystems,Inc.-1.7.0_272-b10 with JDK Azul Systems, Inc.-1.7.0_272-b10 generated 0 new + 6 unchanged - 2 fixed = 6 total (was 8)
+1 💚	findbugs	4m 37s	the patch passed
		_ Other Tests _
-1 ❌	unit	150m 16s	hbase-server in the patch failed.
+1 💚	asflicense	0m 29s	The patch does not generate ASF License warnings.
		211m 35s

Reason	Tests
Failed junit tests	hadoop.hbase.regionserver.TestSplitTransactionOnCluster

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2975/1/artifact/out/Dockerfile
GITHUB PR	#2975
JIRA Issue	HBASE-25596
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux f81601d65ef1 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2975/out/precommit/personality/provided.sh
git revision	branch-1 / `f9a9148`
Default Java	Azul Systems, Inc.-1.7.0_272-b10
Multi-JDK versions	/usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2975/1/artifact/out/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2975/1/testReport/
Max. process+thread count	3660 (vs. ulimit of 10000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2975/1/console
versions	git=1.9.1 maven=3.0.5 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-02-23T05:26:55Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 38s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 1s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 3 new or modified test files.
		_ branch-1 Compile Tests _
+1 💚	mvninstall	10m 48s	branch-1 passed
+1 💚	compile	1m 8s	branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚	compile	1m 6s	branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚	checkstyle	2m 28s	branch-1 passed
+1 💚	shadedjars	4m 28s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 3s	branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚	javadoc	0m 56s	branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+0 🆗	spotbugs	4m 21s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	4m 18s	branch-1 passed
-0 ⚠️	patch	4m 28s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 48s	the patch passed
+1 💚	compile	0m 59s	the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚	javac	0m 59s	the patch passed
+1 💚	compile	1m 1s	the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚	javac	1m 1s	the patch passed
+1 💚	checkstyle	2m 16s	hbase-server: The patch generated 0 new + 0 unchanged - 10 fixed = 0 total (was 10)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedjars	4m 6s	patch has no errors when building our shaded downstream artifacts.
+1 💚	hadoopcheck	5m 56s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2.
+1 💚	javadoc	0m 33s	the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚	javadoc	0m 43s	the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚	findbugs	3m 9s	the patch passed
		_ Other Tests _
+1 💚	unit	122m 57s	hbase-server in the patch passed.
+1 💚	asflicense	0m 31s	The patch does not generate ASF License warnings.
		178m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2975/3/artifact/out/Dockerfile
GITHUB PR	#2975
JIRA Issue	HBASE-25596
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux 471c6b614fb2 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2975/out/precommit/personality/provided.sh
git revision	branch-1 / `9e9301a`
Default Java	Azul Systems, Inc.-1.7.0_272-b10
Multi-JDK versions	/usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2975/3/testReport/
Max. process+thread count	3693 (vs. ulimit of 10000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2975/3/console
versions	git=1.9.1 maven=3.0.5 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

apurtell · 2021-02-23T17:03:00Z

Unless there is a change in review status I will merge this in a couple of hours.

Fix NPE and avoid permanent unreplicated data due to EOF

505ab71

sandeepvinayak commented Feb 22, 2021

View reviewed changes

apurtell requested changes Feb 22, 2021

View reviewed changes

Addressing comments from @apurtell

8a4492e

sandeepvinayak requested a review from apurtell February 22, 2021 22:56

apurtell approved these changes Feb 22, 2021

View reviewed changes

xcangCRM reviewed Feb 23, 2021

View reviewed changes

Merging from latest

e89795c

xcangCRM approved these changes Feb 23, 2021

View reviewed changes

apurtell merged commit 19fe18e into apache:branch-1 Feb 23, 2021

This was referenced Feb 25, 2021

HBASE-25596: Fix NPE and avoid permanent unreplicated data due to EOF #2987

Merged

HBASE-25596: Fix NPE and avoid permanent unreplicated data due to EOF #2990

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-25596: Fix NPE and avoid permanent unreplicated data due to EOF #2975

HBASE-25596: Fix NPE and avoid permanent unreplicated data due to EOF #2975

sandeepvinayak commented Feb 22, 2021

sandeepvinayak Feb 22, 2021

apurtell Feb 22, 2021

sandeepvinayak Feb 22, 2021

apurtell Feb 22, 2021

xcangCRM Feb 23, 2021

sandeepvinayak Feb 23, 2021

xcangCRM Feb 23, 2021

Apache-HBase commented Feb 23, 2021

Apache-HBase commented Feb 23, 2021

apurtell commented Feb 23, 2021

HBASE-25596: Fix NPE and avoid permanent unreplicated data due to EOF #2975

HBASE-25596: Fix NPE and avoid permanent unreplicated data due to EOF #2975

Conversation

sandeepvinayak commented Feb 22, 2021

sandeepvinayak Feb 22, 2021

Choose a reason for hiding this comment

apurtell Feb 22, 2021

Choose a reason for hiding this comment

sandeepvinayak Feb 22, 2021

Choose a reason for hiding this comment

apurtell Feb 22, 2021

Choose a reason for hiding this comment

xcangCRM Feb 23, 2021

Choose a reason for hiding this comment

sandeepvinayak Feb 23, 2021

Choose a reason for hiding this comment

xcangCRM Feb 23, 2021

Choose a reason for hiding this comment

Apache-HBase commented Feb 23, 2021

Apache-HBase commented Feb 23, 2021

apurtell commented Feb 23, 2021