Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-22923 min version of RegionServer to move system table regions #3438

Merged
merged 5 commits into from
Jul 1, 2021

Conversation

virajjasani
Copy link
Contributor

No description provided.

@virajjasani virajjasani requested review from apurtell and bharathv June 29, 2021 10:08
@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 11m 32s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-0 ⚠️ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ branch-1 Compile Tests _
+1 💚 mvninstall 9m 40s branch-1 passed
+1 💚 compile 0m 40s branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 compile 0m 44s branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚 checkstyle 1m 49s branch-1 passed
+1 💚 shadedjars 3m 5s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 49s branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 javadoc 0m 41s branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+0 🆗 spotbugs 3m 5s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 2s branch-1 passed
_ Patch Compile Tests _
+1 💚 mvninstall 1m 50s the patch passed
+1 💚 compile 0m 41s the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 javac 0m 41s the patch passed
+1 💚 compile 0m 44s the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚 javac 0m 44s the patch passed
-1 ❌ checkstyle 1m 37s hbase-server: The patch generated 4 new + 228 unchanged - 0 fixed = 232 total (was 228)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedjars 2m 50s patch has no errors when building our shaded downstream artifacts.
+1 💚 hadoopcheck 4m 26s Patch does not cause any errors with Hadoop 2.8.5 2.9.2.
+1 💚 javadoc 0m 32s the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 javadoc 0m 41s the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚 findbugs 2m 55s the patch passed
_ Other Tests _
-1 ❌ unit 174m 49s hbase-server in the patch failed.
+1 💚 asflicense 0m 40s The patch does not generate ASF License warnings.
227m 24s
Reason Tests
Failed junit tests hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint
hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/1/artifact/out/Dockerfile
GITHUB PR #3438
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 629130e8e8bc 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-3438/out/precommit/personality/provided.sh
git revision branch-1 / 395eb0c
Default Java Azul Systems, Inc.-1.7.0_272-b10
Multi-JDK versions /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10
checkstyle https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/1/artifact/out/diff-checkstyle-hbase-server.txt
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/1/artifact/out/patch-unit-hbase-server.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/1/testReport/
Max. process+thread count 4214 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/1/console
versions git=1.9.1 maven=3.0.5 findbugs=3.0.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

* "hbase.min.version.move.system.tables" as "2.0.0".
* When operator uses this config, it should be used with care, meaning
* we should be confident that even if user table regions come to RS with
* higher version (that rest of cluster), it would not cause any
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The language here is ambiguous. Better to say something like

When the operator uses this configuration option, any version between the current version and the new value of "hbase.min.version.move.system.tables" does not trigger any region movement. It is assumed the configured range of versions do not require special handling.

This should also be committed to all branches, not just branch-1, for consistent functionality across all future releasing versions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@apurtell Created master branch PR #3439 yesterday. Will create branch-2 PR as well.

*
* @return List of Excluded servers for System table regions.
*/
private List<ServerName> getExcludedServersForSystemTableUnlessAllowed() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we club the logic of both the methods and pass an optional boolean for version check?
The two functions that look very much alike except for the tail part.

@bharathv
Copy link
Contributor

bharathv commented Jun 29, 2021

Also, the "localhost" part is suspicious, why does AM look for a localhost address?

@virajjasani
Copy link
Contributor Author

Also, the "localhost" part is suspicious, why does AM look for a localhost address?

It is BOGUS_SERVER_NAME. AM tries to put it in FAILED_OPEN state after getting one:

  private void processBogusAssignments(Map<ServerName, List<HRegionInfo>> bulkPlan) {
    if (bulkPlan.containsKey(LoadBalancer.BOGUS_SERVER_NAME)) {
      // Found no plan for some regions, put those regions in RIT
      for (HRegionInfo hri : bulkPlan.get(LoadBalancer.BOGUS_SERVER_NAME)) {
        regionStates.updateRegionState(hri, State.FAILED_OPEN);
      }
      bulkPlan.remove(LoadBalancer.BOGUS_SERVER_NAME);
    }
  }

And it is RSGroupBasedLoadBalancer that tries to add it:

          //if not server is available assign to bogus so it ends up in RIT
          if(!assignments.containsKey(LoadBalancer.BOGUS_SERVER_NAME)) {
            assignments.put(LoadBalancer.BOGUS_SERVER_NAME, new ArrayList<HRegionInfo>());
          }
          assignments.get(LoadBalancer.BOGUS_SERVER_NAME).add(region);

and similar logic is available few other places in RSGroupBasedLoadBalancer.

Comment on lines +2598 to +2599
if (checkForMinVersion) {
if (!DEFAULT_MIN_VERSION_MOVE_SYS_TABLES_CONFIG.equals(minVersionToMoveSysTables)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super nit: club the conditions

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the argument checkForMinVersion controls this specific flow, I thought keeping it this way would make it more readable. Thought?

Copy link
Contributor

@bharathv bharathv Jun 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a strong opinion. If you ask me, I like the following version but its subjective. I'm fine with whatever you think is good.

checkForMinVersion = checkForMinVersion && !DEFAULT_MIN_VERSION_MOVE_SYS_TABLES_CONFIG.equals(minVersionToMoveSysTables);
if (checkForMinVersion) {
  ,,,,
}

Copy link
Contributor Author

@virajjasani virajjasani Jul 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love these discussions, always learn a thing or two :)

decisionFactor = decisionFactor && additionalFactors yeah, this is also nice way. I wish we had some standards around this.
For now, let me keep it as is as you don't have strong opinion, I still find this simpler from readability viewpoint:

  if (decisionFactor) {
    if (additionalFactors) {
    }
  }

I think this is why we don't have standards because individuals find different approaches as simpler ones than others (but CPU doesn't care :) )

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 11m 32s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-0 ⚠️ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ branch-1 Compile Tests _
+1 💚 mvninstall 10m 54s branch-1 passed
+1 💚 compile 0m 42s branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 compile 0m 45s branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚 checkstyle 1m 55s branch-1 passed
+1 💚 shadedjars 3m 21s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 49s branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 javadoc 0m 42s branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+0 🆗 spotbugs 3m 16s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 12s branch-1 passed
_ Patch Compile Tests _
+1 💚 mvninstall 2m 2s the patch passed
+1 💚 compile 0m 42s the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 javac 0m 42s the patch passed
+1 💚 compile 0m 47s the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚 javac 0m 47s the patch passed
-1 ❌ checkstyle 1m 48s hbase-server: The patch generated 4 new + 231 unchanged - 0 fixed = 235 total (was 231)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedjars 3m 9s patch has no errors when building our shaded downstream artifacts.
+1 💚 hadoopcheck 5m 0s Patch does not cause any errors with Hadoop 2.8.5 2.9.2.
+1 💚 javadoc 0m 32s the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 javadoc 0m 43s the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚 findbugs 3m 9s the patch passed
_ Other Tests _
-1 ❌ unit 155m 4s hbase-server in the patch failed.
+1 💚 asflicense 0m 32s The patch does not generate ASF License warnings.
210m 58s
Reason Tests
Failed junit tests hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/2/artifact/out/Dockerfile
GITHUB PR #3438
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux eddc05d43a0b 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-agent/workspace/Base-PreCommit-GitHub-PR_PR-3438/out/precommit/personality/provided.sh
git revision branch-1 / 28f36f4
Default Java Azul Systems, Inc.-1.7.0_272-b10
Multi-JDK versions /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10
checkstyle https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/2/artifact/out/diff-checkstyle-hbase-server.txt
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/2/artifact/out/patch-unit-hbase-server.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/2/testReport/
Max. process+thread count 5023 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/2/console
versions git=1.9.1 maven=3.0.5 findbugs=3.0.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 15m 10s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-0 ⚠️ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ branch-1 Compile Tests _
+1 💚 mvninstall 10m 23s branch-1 passed
+1 💚 compile 0m 52s branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 compile 0m 52s branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚 checkstyle 2m 3s branch-1 passed
+1 💚 shadedjars 3m 24s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 48s branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 javadoc 0m 43s branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+0 🆗 spotbugs 3m 16s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 12s branch-1 passed
_ Patch Compile Tests _
+1 💚 mvninstall 2m 2s the patch passed
+1 💚 compile 0m 41s the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 javac 0m 41s the patch passed
+1 💚 compile 0m 47s the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚 javac 0m 47s the patch passed
-1 ❌ checkstyle 1m 50s hbase-server: The patch generated 4 new + 231 unchanged - 0 fixed = 235 total (was 231)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedjars 3m 6s patch has no errors when building our shaded downstream artifacts.
+1 💚 hadoopcheck 4m 46s Patch does not cause any errors with Hadoop 2.8.5 2.9.2.
+1 💚 javadoc 0m 32s the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19
+1 💚 javadoc 0m 43s the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10
+1 💚 findbugs 3m 7s the patch passed
_ Other Tests _
-1 ❌ unit 145m 55s hbase-server in the patch failed.
+1 💚 asflicense 0m 30s The patch does not generate ASF License warnings.
205m 16s
Reason Tests
Failed junit tests hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
hadoop.hbase.security.visibility.TestVisibilityLabelsWithACL
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/3/artifact/out/Dockerfile
GITHUB PR #3438
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 774aa1b1dcb2 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-3438/out/precommit/personality/provided.sh
git revision branch-1 / 28f36f4
Default Java Azul Systems, Inc.-1.7.0_272-b10
Multi-JDK versions /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10
checkstyle https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/3/artifact/out/diff-checkstyle-hbase-server.txt
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/3/artifact/out/patch-unit-hbase-server.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/3/testReport/
Max. process+thread count 4488 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3438/3/console
versions git=1.9.1 maven=3.0.5 findbugs=3.0.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

virajjasani added a commit that referenced this pull request Jul 1, 2021
@virajjasani virajjasani merged commit 7d0a72b into apache:branch-1 Jul 1, 2021
@virajjasani virajjasani deleted the HBASE-22923-branch-1 branch July 1, 2021 10:51
virajjasani added a commit that referenced this pull request Jul 1, 2021
virajjasani added a commit that referenced this pull request Jul 1, 2021
virajjasani added a commit that referenced this pull request Jul 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants