HBASE-22723 Have CatalogJanitor report holes and overlaps; i.e. probl…

…ems it sees when doing its regular scan of hbase:meta

Refactor of CatalogJanitor so it generates a
Report on the state of hbase:meta when it runs. Also
refactor so CJ runs even if RIT (previous it would
punt on running if RIT) so it can generate a 'Report'
on the interval regardless. If RIT, it just doesn't
go on to do the merge/split GC as it used to.

If report finds an issue, dump as a WARN message
to the master log.

Follow-on is to make the Report actionable/available
for the Master to pull when it goes to draw the hbck
UI page (could also consider shipping the Report as
part of ClusterMetrics?)

Adds new, fatter Visitor to CJ, one that generates
Report on each run keeping around more findings as
it runs.

Moved some methods around so class reads better;
previous methods were randomly ordered in the class.

M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
 Make a few handy methods public.

M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfo.java
 Add utility as defaults on the Inteface; i.e. is this the first region
 in table, is it last, does a passed region come next, or does passed
 region overlap this region (added tests for this new stuff).

M hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
 Bugfix... handle case where buffer passed is null.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
 Lots of change, reorg., but mostly adding consistency checking
 to the visitor used scanning hbase:meta on a period and the
 generation of a Report on what the scan has found traversing
 hbase:meta. Added a main so could try the CatalogJanitor against
 a running cluster.

A hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitorCluster.java
 Fat ugly test for CatalogJanitor consistency checking.

M hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java
 Add tests for new functionality in RI.

M hbase-shell/src/main/ruby/hbase/table.rb
 Bug fix for case where meta has a null regioninfo; scan was aborting.

Signed-off-by: Andrew Purtell <[email protected]>
Signed-off-by: Wellington Chevreuil <[email protected]>

hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java

@@ -810,7 +810,7 @@ private static RegionInfo getClosestRegionInfo(Connection connection,
    * Returns the column family used for meta columns.
    * @return HConstants.CATALOG_FAMILY.
    */
-  private static byte[] getCatalogFamily() {
+  public static byte[] getCatalogFamily() {
     return HConstants.CATALOG_FAMILY;
   }
 
@@ -826,7 +826,7 @@ private static byte[] getTableFamily() {
    * Returns the column qualifier for serialized region info
    * @return HConstants.REGIONINFO_QUALIFIER
    */
-  private static byte[] getRegionInfoColumn() {
+  public static byte[] getRegionInfoColumn() {
     return HConstants.REGIONINFO_QUALIFIER;
   }
 
@@ -1049,7 +1049,7 @@ public static RegionInfo getRegionInfo(Result data) {
    * @return An RegionInfo instance or null.
    */
   @Nullable
-  private static RegionInfo getRegionInfo(final Result r, byte [] qualifier) {
+  public static RegionInfo getRegionInfo(final Result r, byte [] qualifier) {
     Cell cell = r.getColumnLatestCell(getCatalogFamily(), qualifier);
     if (cell == null) return null;
     return RegionInfo.parseFromOrNull(cell.getValueArray(),

hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfo.java

@@ -70,6 +70,8 @@
  */
 @InterfaceAudience.Public
 public interface RegionInfo {
+  public static final RegionInfo UNDEFINED =
+      RegionInfoBuilder.newBuilder(TableName.valueOf("__UNDEFINED__")).build();
   /**
    * Separator used to demarcate the encodedName in a region name
    * in the new format. See description on new format above.
@@ -775,4 +777,55 @@ static List<RegionInfo> parseDelimitedFrom(final byte[] bytes, final int offset,
     }
     return ris;
   }
+
+
+  /**
+   * @return True if this is first Region in Table
+   */
+  default boolean isFirst() {
+    return Bytes.equals(getStartKey(), HConstants.EMPTY_START_ROW);
+  }
+
+  /**
+   * @return True if this is last Region in Table
+   */
+  default boolean isLast() {
+    return Bytes.equals(getEndKey(), HConstants.EMPTY_START_ROW);
+  }
+
+  /**
+   * @return True if regions are adjacent, if 'after' next. Does not do tablename compare.
+   */
+  default boolean isNext(RegionInfo after) {
+    return Bytes.equals(getEndKey(), after.getStartKey());
+  }
+
+  /**
+   * @return True if RegionInfo is degenerate... if startKey > endKey.
+   */
+  default boolean isDegenerate() {
+    return !isLast() && Bytes.compareTo(getStartKey(), getEndKey()) > 0;
+  }
+
+  /**
+   * @return True if an overlap in region range. Does not do tablename compare.
+   *   Does not check if <code>other</code> has degenerate range.
+   * @see #isDegenerate()
+   */
+  default boolean isOverlap(RegionInfo other) {
+    int startKeyCompare = Bytes.compareTo(getStartKey(), other.getStartKey());
+    if (startKeyCompare == 0) {
+      return true;
+    }
+    if (startKeyCompare < 0) {
+      if (isLast()) {
+        return true;
+      }
+      return Bytes.compareTo(getEndKey(), other.getStartKey()) > 0;
+    }
+    if (other.isLast()) {
+      return true;
+    }
+    return Bytes.compareTo(getStartKey(), other.getEndKey()) < 0;
+  }
 }

hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java

@@ -1359,7 +1359,7 @@ public static long readAsVLong(final byte [] buffer, final int offset) {
    */
   public static int compareTo(final byte [] left, final byte [] right) {
     return LexicographicalComparerHolder.BEST_COMPARER.
-      compareTo(left, 0, left.length, right, 0, right.length);
+      compareTo(left, 0, left == null? 0: left.length, right, 0, right == null? 0: right.length);
   }
 
   /**