Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-23095 Reuse FileStatus in StoreFileInfo #674

Merged
merged 4 commits into from
Oct 8, 2019

Conversation

karthikhw
Copy link
Contributor

We found this performance issue when taking a snapshot on large MOB table where MOB compaction is disabled more than 3 months, is due to MOB data loss issue HBASE-22075.

@@ -152,7 +166,7 @@ public StoreFileInfo(final Configuration conf, final FileSystem fs, final Path i
*/
public StoreFileInfo(final Configuration conf, final FileSystem fs, final FileStatus fileStatus)
throws IOException {
this(conf, fs, fileStatus.getPath());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, this is the problem...

}

private StoreFileInfo(final Configuration conf, final FileSystem fs, final Path initialPath,
final Long createdTimestamp, final Long size) throws IOException {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about just passing a FileStatus in?

Copy link
Contributor Author

@karthikhw karthikhw Sep 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I initially thought the same Duo like directly passing "FileStatus" in constructor but what I feel that "Path" is also in same constructor and even that Path can be constructed from FileStatus. So that looks to me not right. :) Another important is Path can not be removed from constructor because many callers pass only Path object, not FileStatus.

I am fine Duo if you still want to change?
public StoreFileInfo(final Configuration conf, final FileSystem fs, final Path initialPath, final FileStatus filestatus)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is fine, as the FileStatus could be null. And maybe you could add a @nullable annotation to indicate this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing the individual params Ts and size looks cleaner IMO. Anyways explained why selected that path.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this requires extra boxing/unboxing? And what if we need more parameters in the future...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any way is ok. As this was not in a hot path, even that Autoboxing thing was also ok. That 2 params passing looked more clean from the calling stand point. Anyway not so big concern. This looks clear too.

Copy link
Contributor Author

@karthikhw karthikhw Oct 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @anoopsjohn for the confirmation.

@Apache9 Duo, I already did new commit for your suggestions. Please confirm which one you would like to keep the old or new change?
6f4baae

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its ok.. Keep it as the way u have pushed as of now.. Its ok only.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure @anoopsjohn. Thanks again!!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appreciate Duo @Apache9 if you get some time for review :)

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
💙 reexec 0m 33s Docker mode activated.
_ Prechecks _
💚 dupname 0m 0s No case conflicting files found.
💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
💚 @author 0m 0s The patch does not contain any @author tags.
💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
💚 mvninstall 5m 24s master passed
💚 compile 0m 55s master passed
💚 checkstyle 1m 21s master passed
💚 shadedjars 4m 33s branch has no errors when building our shaded downstream artifacts.
💚 javadoc 0m 37s master passed
💙 spotbugs 4m 6s Used deprecated FindBugs config; considering switching to SpotBugs.
💚 findbugs 4m 5s master passed
_ Patch Compile Tests _
💚 mvninstall 4m 54s the patch passed
💚 compile 0m 53s the patch passed
💚 javac 0m 53s the patch passed
💔 checkstyle 1m 16s hbase-server: The patch generated 6 new + 19 unchanged - 0 fixed = 25 total (was 19)
💚 whitespace 0m 0s The patch has no whitespace issues.
💚 shadedjars 4m 36s patch has no errors when building our shaded downstream artifacts.
💚 hadoopcheck 15m 38s Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚 javadoc 0m 35s the patch passed
💚 findbugs 4m 7s the patch passed
_ Other Tests _
💚 unit 160m 20s hbase-server in the patch passed.
💚 asflicense 0m 31s The patch does not generate ASF License warnings.
216m 42s
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/1/artifact/out/Dockerfile
GITHUB PR #674
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 0974af5462ef 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-674/out/precommit/personality/provided.sh
git revision master / ca0d9f3
Default Java 1.8.0_181
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/1/artifact/out/diff-checkstyle-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/1/testReport/
Max. process+thread count 4805 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/1/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
💙 reexec 3m 46s Docker mode activated.
_ Prechecks _
💚 dupname 0m 0s No case conflicting files found.
💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
💚 @author 0m 0s The patch does not contain any @author tags.
💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
💚 mvninstall 6m 4s master passed
💚 compile 0m 59s master passed
💚 checkstyle 1m 29s master passed
💚 shadedjars 5m 6s branch has no errors when building our shaded downstream artifacts.
💚 javadoc 0m 38s master passed
💙 spotbugs 4m 26s Used deprecated FindBugs config; considering switching to SpotBugs.
💚 findbugs 4m 23s master passed
_ Patch Compile Tests _
💚 mvninstall 5m 28s the patch passed
💚 compile 0m 58s the patch passed
💚 javac 0m 58s the patch passed
💔 checkstyle 1m 29s hbase-server: The patch generated 2 new + 19 unchanged - 0 fixed = 21 total (was 19)
💚 whitespace 0m 0s The patch has no whitespace issues.
💚 shadedjars 5m 2s patch has no errors when building our shaded downstream artifacts.
💚 hadoopcheck 17m 30s Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚 javadoc 0m 35s the patch passed
💚 findbugs 4m 30s the patch passed
_ Other Tests _
💔 unit 259m 48s hbase-server in the patch failed.
💚 asflicense 0m 41s The patch does not generate ASF License warnings.
325m 30s
Reason Tests
Failed junit tests hadoop.hbase.util.TestFromClientSide3WoUnsafe
hadoop.hbase.master.assignment.TestMergeTableRegionsProcedure
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/2/artifact/out/Dockerfile
GITHUB PR #674
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux b7b1f92f219e 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-674/out/precommit/personality/provided.sh
git revision master / 5217618
Default Java 1.8.0_181
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/2/artifact/out/diff-checkstyle-hbase-server.txt
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/2/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/2/testReport/
Max. process+thread count 4879 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/2/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@anoopsjohn anoopsjohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Nice catch and analysis. Good job.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
💙 reexec 2m 5s Docker mode activated.
_ Prechecks _
💚 dupname 0m 0s No case conflicting files found.
💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
💚 @author 0m 0s The patch does not contain any @author tags.
💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
💚 mvninstall 5m 31s master passed
💚 compile 0m 54s master passed
💚 checkstyle 1m 20s master passed
💚 shadedjars 4m 38s branch has no errors when building our shaded downstream artifacts.
💚 javadoc 0m 40s master passed
💙 spotbugs 4m 3s Used deprecated FindBugs config; considering switching to SpotBugs.
💚 findbugs 4m 1s master passed
_ Patch Compile Tests _
💚 mvninstall 4m 49s the patch passed
💚 compile 0m 57s the patch passed
💚 javac 0m 57s the patch passed
💔 checkstyle 1m 18s hbase-server: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
💚 whitespace 0m 0s The patch has no whitespace issues.
💚 shadedjars 4m 37s patch has no errors when building our shaded downstream artifacts.
💚 hadoopcheck 15m 42s Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚 javadoc 0m 36s the patch passed
💚 findbugs 4m 12s the patch passed
_ Other Tests _
💚 unit 160m 32s hbase-server in the patch passed.
💚 asflicense 0m 44s The patch does not generate ASF License warnings.
219m 17s
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/3/artifact/out/Dockerfile
GITHUB PR #674
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 5f28b0bbf8da 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-674/out/precommit/personality/provided.sh
git revision master / 5217618
Default Java 1.8.0_181
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/3/artifact/out/diff-checkstyle-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/3/testReport/
Max. process+thread count 4663 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/3/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
💙 reexec 1m 9s Docker mode activated.
_ Prechecks _
💚 dupname 0m 0s No case conflicting files found.
💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
💚 @author 0m 0s The patch does not contain any @author tags.
💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
💚 mvninstall 5m 54s master passed
💚 compile 0m 58s master passed
💚 checkstyle 1m 30s master passed
💚 shadedjars 5m 4s branch has no errors when building our shaded downstream artifacts.
💚 javadoc 0m 38s master passed
💙 spotbugs 4m 27s Used deprecated FindBugs config; considering switching to SpotBugs.
💚 findbugs 4m 24s master passed
_ Patch Compile Tests _
💚 mvninstall 5m 28s the patch passed
💚 compile 0m 57s the patch passed
💚 javac 0m 57s the patch passed
💚 checkstyle 1m 28s the patch passed
💚 whitespace 0m 0s The patch has no whitespace issues.
💚 shadedjars 5m 2s patch has no errors when building our shaded downstream artifacts.
💚 hadoopcheck 17m 25s Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚 javadoc 0m 34s the patch passed
💚 findbugs 4m 29s the patch passed
_ Other Tests _
💔 unit 248m 7s hbase-server in the patch failed.
💚 asflicense 0m 28s The patch does not generate ASF License warnings.
310m 30s
Reason Tests
Failed junit tests hadoop.hbase.master.TestSplitWALManager
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/4/artifact/out/Dockerfile
GITHUB PR #674
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 71e4e1221f58 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-674/out/precommit/personality/provided.sh
git revision master / 2ebdcbc
Default Java 1.8.0_181
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/4/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/4/testReport/
Max. process+thread count 5080 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/4/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

@karthikhw
Copy link
Contributor Author

@Apache9 Can you please approve? It still be point "change requested".

Copy link
Contributor

@Apache9 Apache9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@Apache9 Apache9 merged commit ff52994 into apache:master Oct 8, 2019
asfgit pushed a commit that referenced this pull request Oct 8, 2019
Signed-off-by: Anoop Sam John <[email protected]>
Signed-off-by: Duo Zhang <[email protected]>
asfgit pushed a commit that referenced this pull request Oct 8, 2019
Signed-off-by: Anoop Sam John <[email protected]>
Signed-off-by: Duo Zhang <[email protected]>
asfgit pushed a commit that referenced this pull request Oct 8, 2019
Signed-off-by: Anoop Sam John <[email protected]>
Signed-off-by: Duo Zhang <[email protected]>
infraio pushed a commit to infraio/hbase that referenced this pull request Aug 17, 2020
Signed-off-by: Anoop Sam John <[email protected]>
Signed-off-by: Duo Zhang <[email protected]>
symat pushed a commit to symat/hbase that referenced this pull request Feb 17, 2021
Signed-off-by: Anoop Sam John <[email protected]>
Signed-off-by: Duo Zhang <[email protected]>
(cherry picked from commit 66506a6)

Change-Id: Idd7edd87d0d29024518b8928c1684106ec4f8cc3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants