HBASE-23705 Add CellComparator to HFileContext #1062
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Patch looks big but core changes are small just having cellcomparator in context only and not all over the place. |
|
💔 -1 overall
This message was automatically generated. |
saintstack
force-pushed
the
HBASE-23705
branch
from
d727089 to
4fc5506
Compare
|
Fix complaints and UTs. |
|
💔 -1 overall
This message was automatically generated. |
saintstack
force-pushed
the
HBASE-23705
branch
from
4fc5506 to
e8a42ce
Compare
|
Fix checkstyle |
saintstack
force-pushed
the
HBASE-23705
branch
from
e8a42ce to
f45287b
Compare
|
Undid mistaken setting of encoding and blooms on meta schema. Thats for HBASE-21065. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
Review if anyone has a chance. Thanks. |
HorizonNet
reviewed
Jan 19, 2020
| } else { | ||
| // This is awful, we copy the row array if offheap just so we can do a compare. | ||
| // We do this elsewhere too when Cell is backed by an offheap ByteBuffer. | ||
| // Needs fixing. TODO. |
There was a problem hiding this comment.
Is there already a Jira for this?
There was a problem hiding this comment.
No.
Since making this comment, I see that we do this whenever the BB is offheap. Let me make an issue. This issue adds a method to CellComparator that takes a row held in a ByteBuffer. We need more of this with methods in Comparator that can stride through a BB by index so we don't have to copy on heap to compare. The hard part is an implementation that does not insist on two compare methods -- one for array and another for BB. Would be good if could have one compare only.
There was a problem hiding this comment.
Are you sure on this STack. All the impl of the BB vs Array (vice versa) or BB vs BB all happens inside the BBUtils method. There we are clearly ensuring that we either call getXXxOffset() or getXXXPosition() so that we avoide copying onheap. Only when a user tries to do getRowArray on a BB backed cell we try to copy the row part and then do the comparsion. In that case he should also be using getRowOfset() only which would be 0/.
There was a problem hiding this comment.
Only when a user tries to do getRowArray on a BB backed cell we try to copy the row part and then do the comparsion. In that case he should also be using getRowOfset() only which would be
I should have been more specific. Lets make it so no need to copy-to-compare ever.
There was a problem hiding this comment.
Made the comment more mild. Added pointer to BBUtils, that we do here what it does. Added note to fix ever having to copy on heap.
hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java
Outdated
Show resolved
Hide resolved
hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java
Outdated
Show resolved
Hide resolved
hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
Outdated
Show resolved
Hide resolved
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
Outdated
Show resolved
Hide resolved
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
Outdated
Show resolved
Hide resolved
| @@ -1827,7 +1827,8 @@ protected HFileBlock readBlockDataInternal(FSDataInputStream is, long offset, | |||
|
|
|||
| @Override | |||
| public void setIncludesMemStoreTS(boolean includesMemstoreTS) { | |||
| this.fileContext.setIncludesMvcc(includesMemstoreTS); | |||
| this.fileContext = new HFileContextBuilder(this.fileContext). | |||
| withIncludesMvcc(includesMemstoreTS).build(); | |||
There was a problem hiding this comment.
NIT: Most of the time we use a leading ..
There was a problem hiding this comment.
Do we really need to build new HFileContextBuilder and HFileContext to set?
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
Outdated
Show resolved
Hide resolved
| HFile.getWriterFactoryNoCache(conf).withOutputStream(fout).withFileContext(context) | ||
| .withComparator(CellComparatorImpl.COMPARATOR).create(); | ||
| HFile.Writer writer = HFile.getWriterFactoryNoCache(conf). | ||
| withOutputStream(fout).withFileContext(context).create(); |
There was a problem hiding this comment.
NIT: Most of the time we use a leading ..
There was a problem hiding this comment.
I went through the patch and I made all align w/ your suggestions. Thanks @HorizonNet .
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java
Outdated
Show resolved
Hide resolved
|
Thank you for the review @HorizonNet Will put up a patch in a while that implements your nits. |
anoopsjohn
reviewed
Jan 20, 2020
| * @return greater than 0 if leftCell is bigger, less than 0 if rightCell is bigger, 0 if both | ||
| * cells are equal | ||
| */ | ||
| default int compareRows(ByteBuffer row, Cell cell) { |
There was a problem hiding this comment.
The BB passed here contain only the row bytes? The BB is sliced for row alone?
There was a problem hiding this comment.
Yes.
Javadoc tries to make this explicit. Should I add more?
" * @param row ByteBuffer that wraps a row; will read from current position and will reading all
* remaining; will not disturb the ByteBuffer internal state."
| @@ -154,19 +153,6 @@ private int binarySearch(Cell seekCell, boolean seekBefore) { | |||
| } | |||
| } | |||
|
|
|||
| private int compareRows(ByteBuffer row, Cell seekCell) { | |||
There was a problem hiding this comment.
I see. This method is moved to CellComparator. Is that required? This impl here correctly handle things. Within the BBUtils we are handling HBB/DBB. But in this patch, it is trying to handle in CellComparator which is not good IMHO.
Also within this context we know the passed row BB is sliced for the rk bytes alone. Within CellComparator we can not have such assumption. So its better to keep it here only. I believe only this class is having such need for compares. Any other place?
There was a problem hiding this comment.
We can't have encoders doing their own logic figuring Cell Compare else the issue this patch fixes will perpetually haunt us as encoding implementors make the same mistake of around comparators over and over again; i.e. presuming only one comparator in the system or knowing there are two at least but defaulting to user-space comparator since that is what is used 99.9% of the time.
CellComparator already has methods that do row compare. This is just adding one that hosts the row in ByteBuffer instead of an array... an override.
CellComparators need to do more ByteBuffering, not less. CellComparator needs to learn how to do compares w/o making copies of the ByteBuffer content as it currently does in BBUtills if the BB doesn't have an array because comparators don't know how to stride through a ByteBuffer.
bq. Also within this context we know the passed row BB is sliced for the rk bytes alone. Within CellComparator we can not have such assumption.
CellComparator has one method that does this already taking an array.
There was a problem hiding this comment.
I understood ur arg now. Make sense... The MetaCellComparator was doing on heap copy. So ur impl there is ok as of now. May be we can see whether we can avoid some of those. But can be another jira. +1 for doing this way boss.
There was a problem hiding this comment.
Thanks @anoopsjohn
Will put up new patch that implements your suggestions and those of @HorizonNet
| // If no cellComparator specified, make a guess based off tablename. If hbase:meta, then should | ||
| // be the meta table comparator. Comparators are per table. | ||
| this.cellComparator = cellComparator != null? cellComparator: | ||
| this.tableName != null? CellComparatorImpl.getCellComparator(this.tableName): |
There was a problem hiding this comment.
This I like :-) Our best effort to make things work.
| * @return CellComparator to use going off the <code>tableName</code> passed. | ||
| */ | ||
| public static CellComparator getCellComparator(TableName tableName) { | ||
| return getCellComparator(tableName.toBytes()); |
There was a problem hiding this comment.
Can we just use TableName.isMetaTableName(TableName) here? Why to have the indirection of toBytes and then compare bytes?
There was a problem hiding this comment.
Need both. In actual filecontext, it hosts tablename as bytes only -- not as a TableName object. The TableName.toBytes doesn't actually make bytes. TN itself hosts the name in bytes.
Maybe I should be clearer in a comment that no new arrays are being made in this code?
Comment on lines
628
to
641
| if (comparatorClassName.equals(CellComparatorImpl.COMPARATOR.getClass().getName())) { | ||
| return CellComparatorImpl.COMPARATOR; | ||
| } else if (comparatorClassName.equals( | ||
| CellComparatorImpl.META_COMPARATOR.getClass().getName())) { | ||
| return CellComparatorImpl.META_COMPARATOR; | ||
| } |
There was a problem hiding this comment.
With out this also (with below way of get class based on the name) it will work? U just added a short circuit here? Or u fixing some issue here with below getComparatorClass(comparatorClassName)?
There was a problem hiding this comment.
I think it is to avoid the reflections call if possible
There was a problem hiding this comment.
Just to avoid the reflection in near all cases.
| @@ -319,7 +312,7 @@ public Writer create() throws IOException { | |||
| LOG.debug("Unable to set drop behind on {}", path.getName()); | |||
| } | |||
| } | |||
| return new HFileWriterImpl(conf, cacheConf, path, ostream, comparator, fileContext); | |||
| return new HFileWriterImpl(conf, cacheConf, path, ostream, fileContext); | |||
There was a problem hiding this comment.
So here the comparator class name will be passed within the fileContext from upper layers. Good.
There was a problem hiding this comment.
Yes sir. Idea is that there is one place to get comparator to use. Before this patch, comparator was passed down most of the time but in encoder context and elsewhere in a few locations, comparator was hardcoded because not passed from above.
| @@ -1827,7 +1827,8 @@ protected HFileBlock readBlockDataInternal(FSDataInputStream is, long offset, | |||
|
|
|||
| @Override | |||
| public void setIncludesMemStoreTS(boolean includesMemstoreTS) { | |||
| this.fileContext.setIncludesMvcc(includesMemstoreTS); | |||
| this.fileContext = new HFileContextBuilder(this.fileContext). | |||
| withIncludesMvcc(includesMemstoreTS).build(); | |||
There was a problem hiding this comment.
Do we really need to build new HFileContextBuilder and HFileContext to set?
| @@ -380,7 +386,8 @@ private HFileContext createHFileContext(Path path, | |||
| HFileContextBuilder builder = new HFileContextBuilder() | |||
| .withHBaseCheckSum(true) | |||
| .withHFileName(path.getName()) | |||
| .withCompression(trailer.getCompressionCodec()); | |||
| .withCompression(trailer.getCompressionCodec()) | |||
| .withCellComparator(trailer.createComparator(trailer.getComparatorClassName())); | |||
There was a problem hiding this comment.
Better to have a trailer.getComparator() which internally do this create?
There was a problem hiding this comment.
I want to keep context immutable so it is safe to use in any context. It is why I removed the setter methods. I think it fine building a new one here rather than reuse. Its once per file reader open. Seems fine.
Let me add your getCellComparator suggestion.
There was a problem hiding this comment.
Hmm... Its a static. createComparator is probably right name here...
ramkrish86
reviewed
Jan 20, 2020
| @@ -131,8 +131,7 @@ private int binarySearch(Cell seekCell, boolean seekBefore) { | |||
| int comp = 0; | |||
| while (low <= high) { | |||
| mid = low + ((high - low) >> 1); | |||
| ByteBuffer row = getRow(mid); | |||
| comp = compareRows(row, seekCell); | |||
| comp = this.cellComparator.compareRows(getRow(mid), seekCell); | |||
There was a problem hiding this comment.
I too agree to @anoopsjohn . Seems we deliberately left this compareRows() here. Is there a similar compareRows in every encoder?
There was a problem hiding this comment.
There is a similar array-based compare rows in CellComparator.
See above for my argument that encoders should not each have to do their own figuring of cell compare.
Notice how much cleaner the code in here is now?
ramkrish86
reviewed
Jan 20, 2020
| break; | ||
| case ROWCOL: | ||
| bloomContext = new RowColBloomContext(generalBloomFilterWriter, comparator); | ||
| bloomContext = | ||
| new RowColBloomContext(generalBloomFilterWriter, fileContext.getCellComparator()); |
There was a problem hiding this comment.
So every where it comes from the context only including for the blooms
There was a problem hiding this comment.
Yes sir. One place only instead of hardcodings and guesses at which to use.
|
@ramkrish86 and @anoopsjohn Thanks for the reviews. Helps. You object to the reshuffle suggesting that the encoder keep its comparator manipulations local. I tried that initially but when it meant trying to figure which cellcomparator when, and then casting the comparator and trying to figure how to do meta cell comparator when no current support, and then the thought that any encoder that needs to change dependent on the table they are running against would have to do the same thing, it made me take pause and do this more fundamental change. Let me know if my argument does not convince and I'll just hack up something that works locally for this one encoder. Thanks. |
saintstack
force-pushed
the
HBASE-23705
branch
from
f45287b to
cc691c7
Compare
|
Address review comments. Ran through files I touched with checkstyle. Looking for a +1 please. Thanks. |
anoopsjohn
approved these changes
Jan 21, 2020
|
💔 -1 overall
This message was automatically generated. |
saintstack
force-pushed
the
HBASE-23705
branch
from
cc691c7 to
3f229b1
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Codecs don't have access to what CellComparator to use. Backfill. M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java Adds a new compareRows with default implementation that takes a ByteBuffer. Needed by the index in a block encoder implementation. M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java Adds implementation for meta of new compareRows method. Adds utility method for figuring comparator based off tablename. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java Comparator is in context. Remove redundant handling. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java Comparator is in context. Remove redundant handling. Clean javadoc. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java Clean javadoc. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java Cache context so can use it to get comparator to use later. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java Cache cellcomparator to use. Javdoc on diff between HFileContext and HFileInfo. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java Add CellComparator M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java Remove comparator caching. Get from context instead. M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java Skip a reflection if we can. M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java Javadoc. Removed unused filed.
saintstack
force-pushed
the
HBASE-23705
branch
from
3f229b1 to
4ffef98
Compare
|
The checkstyle complaints are two. One is an old method that is > 150 limit. It is 242 currently (after I tried cutting it down). Leaving it. Not related. Other is an import order I finally figured. On unit tests, I tried them locally and they fail; a checkstyle change suggesting class be final broke mockito. Put up new patch. Thanks for +1 @anoopsjohn |
|
💔 -1 overall
This message was automatically generated. |
|
Checkstyle failure is for the method that is too long (expected). The failed test passes locally. Let me rerun to make sure. |
|
💔 -1 overall
This message was automatically generated. |
|
Unit test is known issue (HBASE-23722). The checkstyle is the method-too-large. Merging. |
asfgit
pushed a commit
that referenced
this pull request
Jan 23, 2020
Codecs don't have access to what CellComparator to use. Backfill. M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java Adds a new compareRows with default implementation that takes a ByteBuffer. Needed by the index in a block encoder implementation. M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java Adds implementation for meta of new compareRows method. Adds utility method for figuring comparator based off tablename. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java Comparator is in context. Remove redundant handling. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java Comparator is in context. Remove redundant handling. Clean javadoc. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java Clean javadoc. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java Cache context so can use it to get comparator to use later. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java Cache cellcomparator to use. Javdoc on diff between HFileContext and HFileInfo. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java Add CellComparator M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java Remove comparator caching. Get from context instead. M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java Skip a reflection if we can. M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java Javadoc. Removed unused filed. Signed-off-by: Anoop Sam John <[email protected]> Signed-off-by: Ramkrishna <[email protected]> Signed-off-by: Jan Hentschel <[email protected]>
taklwu
added a commit
to taklwu/hbase
that referenced
this pull request
Jan 26, 2020
* HBASE-23136 (#712) * HBASE-23136 PartionedMobFileCompactor bulkloaded files shouldn't get replicated (addressing buklload replication related issue raised in HBASE-22380) Signed-off-by: Josh Elser <[email protected]> * HBASE-23042 Parameters are incorrect in procedures jsp (#728) * HBASE-23172 HBase Canary region success count metrics reflect column family successes, not region successes * HBASE-23012 Add HBase 1.3.6 to the downloads page (#738) * HBASE-22679 : Revamping CellUtil (#735) * HBASE-22679 : Revamping CellUtil * checkstyle fix * incorporating review * minor indentation * HBASE-22460 : Reopen regions with very high Store Ref Counts (#600) Signed-off-by Anoop Sam John <[email protected]> * HBASE-15519 Add per-user metrics with lossy counting Introducing property hbase.regionserver.user.metrics.enabled(Default:true) to disable user metrics in case it accounts for any performance issues Close #661 Signed-off-by: Josh Elser <[email protected]> * HBASE-23207 Log a region open journal (#751) Signed-off-by: Abhishek Singh Chouhan <[email protected]> * HBASE-23073 Add an optional costFunction to balance regions according to a capacity rule (#677) Signed-off-by: Wellington Chevreuil <[email protected]> * HBASE-23012 Addendum - Update 1.3.6 Release date in the downloads page * HBASE-23203 NPE in RSGroup info (#747) Signed-off-by: Duo Zhang <[email protected]> * HBASE-23194 Remove unused methods from TokenUtil (#737) Signed-off-by: Peter Somogyi <[email protected]> * HBASE-22991 add HBase 1.4.11 to the downloads page. closes #756 Signed-off-by: Jan Hentschel <[email protected]> * HBASE-23181 Blocked WAL archive: "LogRoller: Failed to schedule flush of XXXX, because it is not online on us" (#753) Signed-off-by: Lijin Bin <[email protected]> Signed-off-by: stack <[email protected]> * HBASE-23216 Add 2.2.2 to download page (#758) Signed-off-by: Guanghao Zhang <[email protected]> * HBASE-15519 Add per-user metrics with lossy counting (addendum) * HBASE-23222 MOB compaction supportability improvements * better logging on MOB compaction process * HFileCleanerDelegate to optionally halt removal of mob hfiles * use archiving when removing committed mob file after bulkload ref failure closes #763 Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Balazs Meszaros <[email protected]> * HBASE-23199 Error populating Table-Attribute fields (#741) Signed-off-by: GuangxuCheng <[email protected]> * HBASE-23187 Update parent region state to SPLIT in meta (#732) * HBASE-23187 Update parent region state to SPLIT in meta * Revert "HBASE-23194 Remove unused methods from TokenUtil (#737)" This reverts commit d7b90b319908113bb90ae871cf4a5843bbf6bbaa. * HBASE-23208 Unit formatting in Master & RS UI Signed-off-by: binlijin <[email protected]> Signed-off-by: Sean Busbey <[email protected]> * HBASE-23227 Upgrade jackson-databind to 2.9.10.1 Signed-off-by: Sean Busbey <[email protected]> * HBASE-20827 Use backoff on CallQueueTooBigException when reporting region state transition Signed-off-by: Josh Elser <[email protected]> * HBASE-23191 EOFE log spam (#733) Convert log message added for 2.2.0 from INFO to DEBUG. Signed-off-by: stack <[email protected]> * HBASE-22917 Proc-WAL roll fails saying someone else has already created log (#544) Signed-off-by: stack <[email protected]> * HBASE-23192 CatalogJanitor consistencyCheck does not log problematic row on exception (#734) Adds logging of row and complaint if consistency check fails during CJ checking. Adds a few more null checks. Does edit on the 'HBCK Report' top line. Signed-off-by: Reid Chan <[email protected]> * HBASE-22739 ArrayIndexOutOfBoundsException when balance (#729) Signed-off-by: stack <[email protected]> * HBASE-23175 Yarn unable to acquire delegation token for HBase Spark jobs * HBASE-23231 ReplicationSource do not update metrics after refresh (#778) Signed-off-by: stack <[email protected]> Signed-off-by: Duo Zhang <[email protected]> * HBASE-23231 Addendum remove redundant metrics.clear in ReplicationSource This is only a problem on master which was introduced when we merge sync replication feature branch back * HBASE-23184 The HeapAllocation in WebUI is not accurate (#730) Signed-off-by: stack <[email protected]> * HBASE-23221 Polish the WAL interface after HBASE-23181 (#774) Removes the closeRegion flag added by HBASE-23181 and instead relies on reading meta WALEdit content. Modified how qualifier is written when the meta WALEdit is for a RegionEventDescriptor so the 'type' is added to the qualifer so can figure type w/o having to deserialize protobuf value content: e.g. HBASE::REGION_EVENT::REGION_CLOSE Added doc on WALEdit and tried to formalize the 'meta' WALEdit type and how it works. Needs complete redo in part as suggested by HBASE-8457. Meantime, some doc and cleanup. Also changed the LogRoller constructor to remove redundant param. Because of constructor change, need to change also TestFailedAppendAndSync, TestWALLockup, TestAsyncFSWAL & WALPerformanceEvaluation.java Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Lijin Bin <[email protected]> * Revert "HBASE-22917 Proc-WAL roll fails saying someone else has already created log (#544)" This reverts commit 538a4c51ff8464f57e26cc22f9c559c1a30b5864. * HBASE-23238 Additional test and checks for null references on ScannerCallableWithReplicas (#780) Signed-off-by: Sean Busbey <[email protected]> (cherry picked from commit 577db5d7e50c56b4773c9ce92b807aae80bf5706 - Test only, removed changes from no more exisiting ScannerCallableWithReplicas class) * HBASE-23244 NPEs running Canary (#784) Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23244 NPEs running Canary (#784) Addendum to fix findbugs complaint. * HBASE-23241 TestExecutorService sometimes fail (#782) Signed-off-by: Wellington Chevreuil <[email protected]> * HBASE-23247 [hbck2] Schedule SCPs for 'Unknown Servers' (#791) Signed-off-by: Sean Busbey <[email protected]> Signed-off-by: Duo Zhang <[email protected]> * HBASE-23243 [pv2] Filter out SUCCESS procedures; on decent-sized cluster, plethora overwhelms problems (#790) Signed-off-by: GuangxuCheng <[email protected]> Signed-off-by: Sean Busbey <[email protected]> * HBASE-23212 : Dynamically reload configs for Region Recovery chore (#773) * HBASE-23212 : Dynamically reload configs for Region Recovery chore * remove redundant volatile * HBASE-23250 Log message about CleanerChore delegate initialization should be at INFO Signed-off-by: Jan Hentschel <[email protected]> * HBASE-21458 Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty Signed-off-by: Sean Busbey <[email protected]> * HBASE-23085 Network and Data related Actions Add monkey actions: - manipulate network packages with tc (reorder, loose,...) - add CPU load - fill the disk - corrupt or delete regionserver data files Extend HBaseClusterManager to allow sudo calls. Signed-off-by: Josh Elser <[email protected]> Signed-off-by: Balazs Meszaros <[email protected]> * HBASE-22980 HRegionPartioner getPartition() method incorrectly partitions the regions of the table. (#590) Signed-off-by: Guangxu Cheng <[email protected]> * HBASE-23230 Enforce member visibility in HRegionServer (#775) * Clean up a bunch of private variable leakage into other classes. Reduces visibility as much as possible, providing getters where access remains necessary or making use of getters that already exist. There remains an insidious relationship between `HRegionServer` and `RSRpcServices`. * Rename `fs` to `dataFs`, `rootDir` as `dataRootDir` so as to distinguish from the new `walFs`, `walRootDir` (and make it easier to spot bugs). * Cleanup or delete a bunch of lack-luster javadoc comments. * Delete a handful of methods that are unused according to static analysis. * Reduces the warning count as reported by IntelliJ from 100 to 7. Signed-off-by: stack <[email protected]> * HBASE-23272 Fix link in Developer guide to "code review checklist" (#805) Signed-off-by: stack <[email protected]> * HBASE-22480 Get block from BlockCache once and return this block to BlockCache twice make ref count error. * HBASE-23262 Cannot load Master UI (#799) Signed-off-by: Duo Zhang <[email protected]> * HBASE-23263 NPE in Quotas.jsp (#800) Signed-off-by: Guangxu Cheng <[email protected]> * HBASE-23257: Track clusterID in stand by masters (#798) This patch implements a simple cache that all the masters can lookup to serve cluster ID to clients. Active HMaster is still responsible for creating it but all the masters will read it from fs to serve clients. RPCs exposing it will come in a separate patch as a part of HBASE-18095. Signed-off-by: Andrew Purtell <[email protected]> Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Guangxu Cheng <[email protected]> * HBASE-23236 Upgrade to yetus 0.11.1 Signed-off-by: stack <[email protected]> * HBASE-22888 Share some stuffs with the initial reader when new stream reader created (#581) Signed-off-by: Duo Zhang <[email protected]> * HBASE-23268 Remove disable/enable ops from doc when altering schema Signed-off-by: Wellington Chevreuil <[email protected]> * HBASE-23228 Allow for jdk8 specific modules on branch-1 in precommit/nightly testing (#804) Signed-off-by: Josh Elser <[email protected]> * HBASE-18439 Subclasses of o.a.h.h.chaos.actions.Action all use the same logger Signed-off-by: Jan Hentschel <[email protected]> Signed-off-by: Guangxu Cheng <[email protected]> * HBASE-23251 - Add Column Family and Table Names to HFileContext and use in HFileWriterImpl logging (#796) Signed-off-by: Andrew Purtell <[email protected]> Signed-off-by: Xu Cang <[email protected]> Signed-off-by: Zheng Hu <[email protected]> * HBASE-23245 : MutableHistogram constructor changes and provide HistogramImpl maxExpected as long (#787) Signed-off-by: Andrew Purtell <[email protected]> Signed-off-by: Xu Cang <[email protected]> Signed-off-by: Guangxu Cheng <[email protected]> * HBASE-23284 Fix Hadoop wiki link in Developer guide to "Distributions and Commercial Support" Signed-off-by: Nick Dimiduk <[email protected]> * HBASE-23290 shell processlist command is broken * HBASE-23283 Provide clear and consistent logging about the period of enabled chores Signed-off-by: Sean Busbey <[email protected]> * HBASE-19450 Add log about average execution time for ScheduledChore Signed-off-by: Sean Busbey <[email protected]> * HBASE-23294 ReplicationBarrierCleaner should delete all the barriers for a removed region which does not belong to any serial replication peer (#827) Signed-off-by: stack <[email protected]> * HBASE-19450 Addendum Limit logging of chore execution time at INFO to once per 5 minutes. * Ensure MovingAverage related classes are IA.Private * Move trace logging into MovingAverage class Signed-off-by: Duo Zhang <[email protected]> * HBASE-23102: Improper Usage of Map putIfAbsent (#828) Signed-off-by: Wellington Chevreuil <[email protected]> * HBASE-22969 A new binary component comparator(BinaryComponentComparator) to perform comparison of arbitrary length and position (#829) Signed-off-by: Balazs Meszaros <[email protected]> * HBASE-23289 Update links to Hadoop wiki in book and code Signed-off-by: Nick Dimiduk <[email protected]> * HBASE-23282 HBCKServerCrashProcedure for 'Unknown Servers' Have the existing scheduleRecoveries launch a new HBCKSCP instead of SCP. It gets regions to recover from Master in-memory context AND from a scan of hbase:meta. This new HBCKSCP is For processing 'Unknown Servers', servers that are 'dead' and purged but still have references in hbase:meta. Rare occurance but needs tooling to address. Later have catalogjanitor take care of these deviations between Master in-memory and hbase:meta content (usually because of overdriven cluster with failed RPCs to hbase:meta, etc) Changed expireServers in ServerManager so could pass in custom reaction to expired server.... This is how we run our custom HBCKSCP while keeping all other aspects of expiring services (rather than try replicate it externally). * HBASE-23182 The create-release scripts are broken (#736) Signed-off-by: stack <[email protected]> Signed-off-by: Bharath Vissapragada <[email protected]> * HBASE-23318 LoadTestTool doesn't start (#848) * Package the test jar from hbase-zookeeper into lib/ Signed-off-by: Duo Zhang <[email protected]> * HBASE-23278 Add a table-level compaction progress display on the UI (#816) Signed-off-by: Guangxu Cheng <[email protected]> * HBASE-23315 Miscellaneous HBCK Report page cleanup * Add a bit of javadoc around SerialReplicationChecker. * Miniscule edit to the profiler jsp page and then a bit of doc on how to make it work that might help. * Add some detail if NPE getting BitSetNode to help w/ debug. * Change HbckChore to log region names instead of encoded names; helps doing diagnostics; can take region name and query in shell to find out all about the region according to hbase:meta. * Add some fix-it help inline in the HBCK Report page – how to fix. * Add counts in procedures page so can see if making progress; move listing of WALs to end of the page. * HBASE-23308: Review of NullPointerExceptions (#836) Signed-off-by: stack <[email protected]> * HBASE-22607. TestExportSnapshotNoCluster fails intermittently * HBASE-23321 [hbck2] fixHoles of fixMeta doesn't update in-memory state * HBASE-23259: Populate master address end points in cluster/rs configs (#807) All the clients need to know the master RPC end points while using master based registry for creating cluster connections. This patch amends the test cluster utility to populate these configs in the base configuration object used to spin up the cluster. The config key added here ("hbase.master.addrs") is used in the subsequent patches for HBASE-18095. Signed-off-by: Nick Dimiduk <[email protected]> * HBASE-23322 [hbck2] Simplification on HBCKSCP scheduling Signed-off-by: Lijin Bin <[email protected]> * HBASE-23223 Support the offsetLock of bucketCache to use strong ref (#764) Signed-off-by: Wellington Chevreuil <[email protected]> * HBASE-23234 Provide .editorconfig based on checkstyle configuration (#846) This file is generated using IntelliJ, following these steps: #. Open Preferences > Editor > Code Style #. Select (config wheel) > Import Schema > CheckStyle Configuration #. Select `hbase-checkstyle/src/main/resources/hbase/checkstyle.xml` #. Select (config wheel) > Export > EditorConfig File Signed-off-by: Sean Busbey <[email protected]> * HBASE-23237 Prevent Negative values in metrics requestsPerSecond Closes #834 Signed-off-by: Guangxu Cheng <[email protected]> Signed-off-by: Josh Elser <[email protected]> * HBASE-23325 [UI]rsgoup average load keep two decimals (#860) Signed-off-by: Guangxu Cheng <[email protected]> * HBASE-23328 info:regioninfo goes wrong when region replicas enabled (#863) Signed-off-by: Ramkrishna <[email protected]> Signed-off-by: Guangxu Cheng <[email protected]> * HBASE-23329 Remove unused methods from RequestConverter (#870) Signed-off-by: Sean Busbey <[email protected]> * HBASE-23307 Add running of ReplicationBarrierCleaner to hbck2 fixMeta invocation (#859) Signed-off-by: Lijin Bin <[email protected]> * HBASE-22969 A new binary component comparator(BinaryComponentComparator) to perform comparison of arbitrary length and position; ADDENDUM (#869) Signed-off-by: Sean Busbey <[email protected]> * HBASE-23085 Network and Data related Actions; ADDENDUM (#871) Fix percentage in String.format Signed-off-by: Sean Busbey <[email protected]> * HBASE-23197 'IllegalArgumentException: Wrong FS' on edits replay when… (#740) Signed-off-by: Josh Elser <[email protected]> * HBASE-23334 The table-lock node of zk is not needed since HBASE-16786 (#873) Signed-off-by: Guangxu Cheng <[email protected]> * HBASE-23312 HBase Thrift SPNEGO configs (HBASE-19852) should be backwards compatible HBase Thrift SPNEGO configs should not be required. The `hbase.thrift.spnego.keytab.file` and `hbase.thrift.spnego.principal` configs should fall back to the `hbase.thrift.keytab.file` and `hbase.thrift.kerberos.principal` configs. This will avoid any issues during upgrades. Signed-off-by: Josh Elser <[email protected]> Amending-author: Josh Elser <[email protected]> Closes #850 * HBASE-23293 [REPLICATION] make ship edits timeout configurable (#825) Signed-off-by: Guangxu Cheng <[email protected]> Signed-off-by: Wellington Chevreuil <[email protected]> * HBASE-23336 [CLI] Incorrect row(s) count 'clear_deadservers' (#875) Signed-off-by: Guangxu Cheng <[email protected]> Signed-off-by: Lijin Bin <[email protected]> * HBASE-20395 Displaying thrift server type on the thrift page (#811) Signed-off-by: Lijin Bin <[email protected]> Signed-off-by: Bharath Vissapragada <[email protected]> * HBASE-23117: Bad enum in hbase:meta info:state column can fail loadMeta and stop startup (#867) * Handling the BAD value in info:state columns in hbase:meta * Adding a unit test and region encoded name in the LOG * Adding a null check for region state to complete the test scenario and fixing the nit Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: GuangxuCheng <[email protected]> Signed-off-by: stack <[email protected]> * HBASE-23313 [hbck2] setRegionState should update Master in-memory sta… (#864) Signed-off-by: Mingliang Liu <[email protected]> Signed-off-by: stack <[email protected]> * HBASE-20395 Addendum Displaying thrift server type on the thrift page * HBASE-23323 Update downloads page for Apache HBase 1.4.12. (#886) Signed-off-by: stack <[email protected]> * HBASE-23352: Allow chaos monkeys to access cmd line params, and improve FillDiskCommandAction (#885) Instead of using the default properties when checking for monkey properties, now we use the ones already extended with command line params. Change FillDiskCommandAction to try to stop the remote process if the command failed with an exception. Signed-off-by: stack <[email protected]> * HBASE-23342 : Handle NPE while closing compressingStream (#877) Signed-off-by Anoop Sam John <[email protected]> * HBASE-23298 Refactor LogRecoveredEditsOutputSink and BoundedLogWriterCreationOutputSink (#832) Signed-off-by: Duo Zhang <[email protected]> * HBASE-23335 : Improving cost functions array copy in StochasticLoadBalancer (#874) * HBASE-23337 Release scripts should rely on maven for deploy. (#887) - switch to nexus-staging-maven-plugin for asf-release - refactor release-build to use mvn deploy and its output. - cleaned up some tabs in the root pom Signed-off-by: stack <[email protected]> * HBASE-23345 Table need to replication unless all of cfs are excluded (#881) Signed-off-by: stack <[email protected]> Signed-off-by: Guanghao Zhang <[email protected]> * HBASE-22096 /storeFile.jsp shows CorruptHFileException when the storeFile is a reference file (#888) Signed-off-by: Lijin Bin <[email protected]> * HBASE-23356 When construct StoreScanner throw exceptions it is possible to left some KeyValueScanner not closed. (#891) Signed-off-by: GuangxuCheng <[email protected]> * HBASE-23357 Add 2.1.8 to download page (#892) Signed-off-by: Guanghao Zhang <[email protected]> * HBASE-23362: [WalPrettyPrinter] print/filter by table name. (#898) Signed-off-by: Wellington Chevreuil <[email protected]> * HBASE-22529 Add sanity check for in-memory compaction policy Signed-off-by: Toshihiro Suzuki <[email protected]> * HBASE-23365 Minor change MemStoreFlusher's log (#900) Signed-off-by: GuangxuCheng <[email protected]> Signed-off-by: Xu Cang <[email protected]> * HBASE-23361 Limit two decimals in total average load (#897) Signed-off-by: GuangxuCheng <[email protected]> Signed-off-by: Xu Cang <[email protected]> * HBASE-23367 Remove unused constructor from WALPrettyPrinter (#901) Signed-off-by: Lijin Bin <[email protected]> Signed-off-by: Xu Cang <[email protected]> * HBASE-23364 HRegionServer sometimes does not shut down. * HBASE-23373 Log `RetriesExhaustedException` context with full time precision (#903) Signed-off-by: Lijin Bin <[email protected]> * HBASE-23309: Adding the flexibility to ChainWalEntryFilter to filter the whole entry if all cells get filtered (#837) HBASE-23309: Adding the flexibility to ChainWalEntryFilter to filter the whole entry if all cells get filtered * HBASE-23303 Add security headers to REST server/info page (#843) Signed-off-by: Toshihiro Suzuki <[email protected]> Signed-off-by: Sean Busbey <[email protected]> * HBASE-22280 Separate read/write handler for priority request(especial… (#202) Signed-off-by: Yu Li <[email protected]> * HBASE-18382 add transport type info into Thrift UI (#880) Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Bharath Vissapragada <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23377 Balancer should skip disabled tables's regions (#908) Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: GuangxuCheng <[email protected]> Signed-off-by Anoop Sam John <[email protected]> Signed-off-by: Reid Chan <[email protected]> * HBASE-23066 Allow cache on write during compactions when prefetching is (#919) Contributed by : Jacob LeBlanc * HBASE-23552 Format Javadocs on ITBLL We have this nice description in the java doc on ITBLL but it's unformatted and thus illegible. Add some formatting so that it can be read by humans. Signed-off-by: Jan Hentschel <[email protected]> Signed-off-by: Josh Elser <[email protected]> * HBASE-23379 Clean Up FSUtil getRegionLocalityMappingFromFS Signed-off-by: Jan Hentschel <[email protected]> * HBASE-23556: Minor ChoreService Cleanup (#927) * HBASE-23553 Snapshot referenced data files are deleted in some case (#922) * HBASE-23360 [CLI] Fix help command 'set_quota' for removing limits (#896) * Revert "HBASE-23066 Allow cache on write during compactions when prefetching is (#919)" TestCacheOnWrite failing all the time. This reverts commit d561130e82c5956f0dd9fff5c6f6240a686d3d6a. * HBASE-23554 Encoded regionname to regionname utility (#923) Adds shell command regioninfo: hbase(main):001:0> regioninfo '0e6aa5c19ae2b2627649dc7708ce27d0' {ENCODED => 0e6aa5c19ae2b2627649dc7708ce27d0, NAME => 'TestTable,,1575941375972.0e6aa5c19ae2b2627649dc7708ce27d0.', STARTKEY => '', ENDKEY => '00000000000000000000299441'} Took 0.4737 seconds Signed-off-by: Sean Busbey <[email protected]> Signed-off-by: Duo Zhang <[email protected]> * HBASE-23554 Encoded regionname to regionname utility (#923); ADDENDUM * HBASE-23555 TestQuotaThrottle is broken (#924) * HBASE-23566: Fix package/packet terminology problem in chaos monkeys (#933) s/package/packet/g Signed-off-by: Sean Busbey <[email protected]> * HBASE-23570 Point users to the async-profiler home page if diagrams are coming up blank (#937) * HBASE-23575 Remove dead code in AsyncRegistry (#929) Removes a bunch of dead code and fixes some checkstyle nits. Signed-off-by: Viraj Jasani <[email protected]> Signed-off-by: Sean Busbey <[email protected]> * HBASE-23380 General cleanup of FSUtil (#912) * Clean up JavaDocs * Clean up logging and error messages * Remove superfluous code * Replace static code with library call * Do not swallow Interrupted Exceptions * Use try-with-resources * User multi-Exception catches to reduce code size Signed-off-by: Jan Hentschel <[email protected]> Signed-off-by: Sean Busbey <[email protected]> * HBASE-22920 github pr testing job should use dev-support script (#883) Signed-off-by: stack <[email protected]> Signed-off-by: Sean Busbey <[email protected]> Signed-off-by: Bharath Vissapragada <[email protected]> * HBASE-23582 Unbalanced braces in string representation of table descriptor Signed-off-by: Lijin Bin <[email protected]> Signed-off-by: Jan Hentschel <[email protected]> * HBASE-23066 Allow cache on write during compactions when prefetching … (#935) * HBASE-23066 Allow cache on write during compactions when prefetching is enabled * Fix checkstyle issues - recommit * HBASE-23065 [hbtop] Top-N heavy hitter user and client drill downs Signed-off-by: Toshihiro Suzuki <[email protected]> Signed-off-by: Josh Elser <[email protected]> Signed-off-by: Andrew Purtell <[email protected]> * HBASE-23575 Remove dead code in AsyncRegistry (addendum) Additions to MiniHBaseCluster needed for branch-2. Signed-off-by: Jan Hentschel <[email protected]> Signed-off-by: Xu Cang <[email protected]> Signed-off-by: Sean Busbey <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23239 Reporting on status of backing MOB files from client-facing cells (#785) * Adds a new MapReduce job that builds a report of health of mob files * Also builds background information on mob system use * add a basic mob architecture in the ref guide to explain how mob takes the reference hfile value and finds the actual cell contents * add a troubleshooting mob section to the ref guide to explain how to do a mob reference scan. Signed-off-by: Peter Somogyi <[email protected]> * HBASE-23549 Document steps to disable MOB for a column family (#928) Signed-off-by: Peter Somogyi <[email protected]> Signed-off-by: Josh Elser <[email protected]> * HBASE-20461 Implement fsync for AsyncFSWAL (#947) Signed-off-by: stack <[email protected]> * HBASE-23376 NPE happens while replica region is moving (#906) Signed-off-by: Duo Zhang <[email protected]> * HBASE-23594 Procedure stuck due to region happen to recorded on two servers. (#953) Signed-off-by: stack <[email protected]> * HBASE-23564 RegionStates may has some expired serverinfo and make regions do not balance. (#930) Signed-off-by: stack <[email protected]> Signed-off-by: Wellington Chevreuil <[email protected]> * HBASE-23572 In 'HBCK Report', distringush between live, dead, and unknown servers Signed-off-by: Sean Busbey <[email protected]> * HBASE-23320 Upgrade surefire plugin to 3.0.0-M4 Signed-off-by: Peter Somogyi <[email protected]> * HBASE-23581 Creating table gets stuck when specifying an invalid split policy as METADATA (#942) Signed-off-by: Lijin Bin <[email protected]> Signed-off-by: Anoop Sam John <[email protected]> Signed-off-by: Xu Cang <[email protected]> * HBASE-23589: FlushDescriptor contains non-matching family/output combinations (#949) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Lijin Bin <[email protected]> * HBASE-23613 ProcedureExecutor check StuckWorkers blocked by DeadServe… (#960) Signed-off-by: stack <[email protected]> Signed-off-by: Duo Zhang <[email protected]> * HBASE-23326 Implement a ProcedureStore which stores procedures in a HRegion (#941) Signed-off-by: Guanghao Zhang <[email protected]> Signed-off-by: stack <[email protected]> * HBASE-23374 ExclusiveMemHFileBlock’s allocator should not be hardcoded as ByteBuffAllocator.HEAP Signed-off-by: stack <[email protected]> Signed-off-by: Jan Hentschel <[email protected]> * HBASE-23238: Remove 'static'ness of cell counter in LimitKVsReturnFilter (addendum) (#963) Having it as static means the test cannot be parameterized (ran into this issue in HBASE-23305). That happens because the field is not reset between parameterized runs. * HBASE-23286 Improve MTTR: Split WAL to HFile (#820) Signed-off-by: Duo Zhang <[email protected]> * HBASE-23619 Used built-in formatting for logger in hbase-zookeeper Signed-off-by: stack <[email protected]> * Adding developer details to pom.xml * HBASE-23617 Add a stress test tool for region based procedure store (#962) Signed-off-by: stack <[email protected]> * HBASE-23621 Reduced the number of Checkstyle violations in tests of hbase-common Signed-off-by: stack <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23622 Reduced the number of Checkstyle violations in hbase-common Signed-off-by: stack <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23618 Add a tool to dump procedure info in the WAL file (#969) Signed-off-by: stack <[email protected]> * HBASE-23618 Addendum add main method * HBASE-23626 Reduced number of Checkstyle violations in tests in hbase-common Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23615 Use a dedicated thread for executing WorkerMonitor in Pro… (#961) Signed-off-by: stack <[email protected]> Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: virajjasani <[email protected]> * HBASE-23590 : Update maxStoreFileRefCount to maxCompactedStoreFileRefCount for auto region recovery based on old reader references Signed-off-by: Anoop Sam John <[email protected]> * HBASE-23625 Reduced number of Checkstyle violations in hbase-common Signed-off-by: stack <[email protected]> * HBASE-23627 Resolved remaining Checkstyle violations in hbase-thrift Signed-off-by: Viraj Jasani <[email protected]> Signed-off-by: stack <[email protected]> * HBASE-23623 Reduced the number of Checkstyle violations in hbase-rest Signed-off-by: stack <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23624 Add a tool to dump the procedure info in HFile (#975) Signed-off-by: stack <[email protected]> * HBASE-23629: Add to 'Supporting Projects' in site (#976) * Update supportingprojects.xml Signed-off-by: Jan Hentschel <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23588 : Cache index & bloom blocks on write if CacheCompactedBlocksOnWrite is enabled Signed-off-by: ramkrish86 <[email protected]> Signed-off-by: chenxu14 <[email protected]> * HBASE-23628: Remove Apache Commons Digest Base64 (#977) Signed-off-by: stack <[email protected]> * HBASE-23596 HBCKServerCrashProcedure can double assign Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Lijin Bin <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> Change its behavior so it will only look in hbase:meta if the call to the super class turns up zero references. Only then will it search hbase:meta for references to 'Unknown Servers'. Normal operation where we read Master context is usual and sufficient. The scan of hbase:meta is only for case where Master state has been corrupted and we need to clear out 'Unknown Servers'. * HBASE-23632 DeadServer cleanup (#979) Signed-off-by: Bharath Vissapragada <[email protected]> * HBASE-23098 [bulkload] If one of the peers in a cluster is configured with NAMESPACE level, its hfile-refs(zk) will be backlogged (#676) Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: stack <[email protected]> * HBASE-23587 The FSYNC_WAL flag does not work on branch-2.x (#974) Signed-off-by: Guanghao Zhang <[email protected]> Signed-off-by: stack <[email protected]> * HBASE-23369 Auto-close 'unknown' Regions reported as OPEN on RegionServers Master force-closes unknown/incorrect Regions OPEN on RS M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java Added a note and small refactor. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java Fix an NPE when CJ ran. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java Minor clean up of log message; make it clearer. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java Make it so closeRegionSilentlyAndWait can be used w/o timeout. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java If a RegionServer Report notes a Region is OPEN and the Master does not know of said Region, close it (We used to crash out the RegionServer) M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateNode.java Minor tweak of toString -- label should be state, not rit (confusing). M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java Doc. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/TransitRegionStateProcedure.java Add region name to exception. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/HBCKServerCrashProcedure.java Be more careful about which Regions we queue up for reassign. This procedure is run by the operator so could happen at any time. We will likely be running this when Master has some accounting of cluster members so check its answers for what Regions were on server before running. M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Doc and we were misrepresenting the case where a Region as not in RIT when we got CLOSE -- we were reporting it as though it was already trying to CLOSE. * HBASE-23333 Include Call.toShortString() in sendCall exceptions * HBASE-23369 Auto-close 'unknown' Regions reported as OPEN on RegionServers; ADDENDUM on master * HBASE-23635 Reduced number of Checkstyle violations in hbase-mapreduce Signed-off-by: Viraj Jasani <[email protected]> Signed-off-by: stack <[email protected]> * HBASE-23588 : Cache index & bloom blocks on write (ADDENDUM) Signed-off-by: Anoop Sam John <[email protected]> * HBASE-23636 Disable table may hang when regionserver stop or abort. (#982) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: virajjasani <[email protected]> * HBASE-23645 Fixed remaining Checkstyle violations in hbase-common tests Signed-off-by: Viraj Jasani <[email protected]> * HBASE-22908 Link To HBase 1.4 Documentation on HBase Site (#993) Signed-off-by: stack <[email protected]> Signed-off-by: Sean Busbey <[email protected]> * HBASE-23651 Region balance throttling can be disabled (#991) Signed-off-by: Viraj Jasani <[email protected]> Signed-off-by: Anoop Sam John <[email protected]> * HBASE-23660 hbase:meta's table.jsp ref to wrong rs address (#999) Signed-off-by: Wellington Chevreuil <[email protected]> * HBASE-23658 Fix flaky TestSnapshotFromMaster (#998) Signed-off-by: Duo Zhang <[email protected]> * HBASE-23663 Allow dot and hyphen in Profiler's URL (#1002) Signed-off-by: Sean Busbey <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23655 Fix flaky TestRSGroupsKillRS: should wait the SCP to finish (#996) Signed-off-by: Duo Zhang <[email protected]> * HBASE-23654 Adding Apache Trafodion and EsgynDB to 'Powered by Apache HBase' page Signed-off-by: Viraj Jasani <[email protected]> Signed-off-by: Jan Hentschel <[email protected]> * HBASE-23664 Upgrade JUnit to 4.13 (#1004) Signed-off-by: Sean Busbey <[email protected]> Signed-off-by: Bharath Vissapragada <[email protected]> Signed-off-by: Jan Hentschel <[email protected]> Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> * HBASE-22285 A normalizer which merges small size regions with adjacent regions (#978) Signed-off-by: Viraj Jasani <[email protected]> Signed-off-by: stack <[email protected]> * HBASE-23378: Clean Up FSUtil setClusterId (#910) Signed-off-by: stack <[email protected]> * HBASE-23055 Alter hbase:meta Make it so hbase:meta can be altered. TableState for hbase:meta was hardcoded ENABLED. Make it dynamic. State is now kept in current active Master. It is transient so falls back to default if Master crashes. Add to registry a getMetaTableState which reads mirrored state from zookeeper (NOT from Master and defaults ENABLED if no implementation or error fetching state). hbase:meta schema will be bootstrapped from the filesystem. Changes to filesystem schema are atomic so we should be ok if Master fails mid-edit (TBD). Undoes a bunch of guards that prevented our being able to edit hbase:meta. TODO: Tests, more clarity around hbase:meta table state, and undoing references to hard-coded hbase:meta regioninfo. M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java Throw illegal access exception if you try to use MetaTableAccessor getting state of the hbase:meta table. M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java Add fetching of hbase:meta table state from registry. Adds cache of tablestates w/ a ttl of 1 second (adjustable). M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java Add querying registry for hbase:meta table state. M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java Add querying of mirrored table state for hbase:meta table. M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java Shutdown access. M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java Just cleanup. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java Add state holder for hbase:meta. Removed unused methods. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java Shut down access. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java Allow hbase:meta to be disabled. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java Allow hbase:meta to be enabled. Signed-off-by: Bharath Vissapragada <[email protected]> * HBASE-23659 BaseLoadBalancer#wouldLowerAvailability should consider region replicas (#1001) Signed-off-by: stack <[email protected]> Signed-off-by: Duo Zhang <[email protected]> * HBASE-23668 Master log start filling with "Flush journal status" messages" This reverts commit fb9fa04da72379431d13f22a7e5d8e75ae1267be. i.e. reapplication of patch that was preamaturely applied. Signed-off-by: Duo Zhang <[email protected]> * Revert "HBASE-23668 Master log start filling with "Flush journal status" messages" Minor addendum fixing log message. * HBASE-23662 : Replace HColumnDescriptor(String cf) with ColumnFamilyDescriptor Signed-off-by: Peter Somogyi <[email protected]> Signed-off-by: Jan Hentschel <[email protected]> * HBASE-23675 Move to Apache parent POM version 22 * HBASE-23165 [hbtop] Some modifications from HBASE-22988 (#987) Signed-off-by: stack <[email protected]> * Revert "HBASE-23055 Alter hbase:meta" This reverts commit 9abdb7b5ae4de0b6d839f5c5d85e9bb899b5edbd. * HBASE-23646 Resolved remaining Checkstyle violations in tests of hbase-rest Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23681 Add UT for procedure store region flusher (#1024) Signed-off-by: stack <[email protected]> * HBASE-23662 : Keep HCD constructor until shell usages are replaced * HBASE-23679 FileSystem objects leak when cleaned up in cleanupBulkLoad The cleanupBulkLoad method is only called for the first Region in the table which was being bulk loaded into. This means that potentially N-1 other RegionServers (where N is the number of RegionServers) will leak one FileSystem object into the FileSystem cache which will never be cleaned up. We need to do this clean-up as a part of secureBulkLoadHFiles otherwise we cannot guarantee that heap usage won't grow unbounded. Closes #1029 Signed-off-by: Sean Busbey <[email protected]> * HBASE-23383 [hbck2] `fixHoles` should queue assignment procedures for any regions its fixing (#917) The current process for an operator, after fixing holes in meta, is to manually disable and enable the whole table. Let's try to avoid bringing the whole table offline if we can. Have the master attempt to queue up assignment procedures for any new regions it creates. Signed-off-by: stack <[email protected]> * HBASE-23687 DEBUG logging cleanup (#1040) Signed-off-by: Jan Hentschel <[email protected]> * HBASE-23689: Bookmark for github PR to jira redirection (#1042) Signed-off-by: stack <[email protected]> * HBASE-23688 Update docs for setting up IntelliJ as a development environment (#1041) Signed-off-by: stack <[email protected]> * HBASE-23569 : Validate that all default chores of HMaster are scheduled Signed-off-by: Andrew Purtell <[email protected]> * HBASE-23691 Add 2.2.3 to download page (#1045) Signed-off-by: Jan Hentschel <[email protected]> * fix 500/NPE of region.jsp (#1033) Signed-off-by: Wellington Chevreuil <[email protected]> * HBASE-23683 Make HBaseInterClusterReplicationEndpoint more extensible (#1027) Signed-off-by: Bharath Vissapragada <[email protected]> Signed-off-by: Josh Elser <[email protected]> Signed-off-by: binlijin <[email protected]> * HBASE-23665: Split unit tests from TestTableName into a separate test-only class. (#1032) Signed-off-by: Nick Dimiduk <[email protected]> * Revert "fix 500/NPE of region.jsp (#1033)" Missed the jira number on that commit message. Will re-apply it with the jira number. This reverts commit d60ce17c1765a445e944738f49953579bdf0bba6. * HBASE-23677 fix 500/NPE of region.jsp (#1033) Signed-off-by: Wellington Chevreuil <[email protected]> * HBASE-23674 Too many rit page Numbers show confusion * HBASE-23694 After RegionProcedureStore completes migration of WALProcedureStore, still running WALProcedureStore.syncThread keeps trying to delete now inexistent log files. (#1048) Signed-off-by: Duo Zhang <[email protected]> * HBASE-23652 Move the unsupported procedure type check before migrating to RegionProcedureStore (#1018) Signed-off-by: Nick Dimiduk <[email protected]> * HBASE-23695 Fail gracefully if no category is present Signed-off-by: Wellington Chevreuil <[email protected]> Closes #1052 * HBASE-23347 Allow custom authentication methods for RPCs Decouple the HBase internals such that someone can implement their own SASL-based authentication mechanism and plug it into HBase RegionServers/Masters. Comes with a design doc in dev-support/design-docs and an example in hbase-examples known as "Shade" which uses a flat-password file for authenticating users. Closes #884 Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Andrew Purtell <[email protected]> Signed-off-by: Reid Chan <[email protected]> * HBASE-23653 Expose content of meta table in web ui (#1020) Adds a display of the content of 'hbase:meta' to the Master's table.jsp, when that table is selected. Supports basic pagination, filtering, &c. Signed-off-by: stack <[email protected]> Signed-off-by: Bharath Vissapragada <[email protected]> * HBASE-23569 : Validate that all default chores of HRegionServer are scheduled (ADDENDUM) Signed-off-by: Andrew Purtell <[email protected]> * HBASE-23703 Add HBase 2.2.3 documentation to website (#1059) Signed-off-by: Jan Hentschel <[email protected]> * HBASE-23701 Try to converge automated checks around Category We have a compile-time guarantee to either have a category array of length zero or one because the category annotation is non-repeatable. Also tries to clean up the checking for unit-test annotations. Closes #1057 Signed-off-by: Viraj Jasani <[email protected]> Signed-off-by: Bharath Vissapragada <[email protected]> * HBASE-23690 Checkstyle plugin complains about our checkstyle.xml format; doc how to resolve mismatched version (#1044) Signed-off-by: Jan Hentschel <[email protected]> Signed-off-by: Bharath Vissapragada <[email protected]> * HBASE-23612 Add new profile to make hbase build success on ARM (#959) * HBASE-23700 Upgrade checkstyle and plugin versions (#1056) Bump checkstyle version to 8.28, maven-checkstyle-plugin to 3.1.0. As per HBASE-23242 and the updated checkstyle docs[1], the LineLength check should be placed under an instance of Checker. [1] https://checkstyle.sourceforge.io/config_sizes.html#LineLength Co-authored-by: Bharath Vissapragada <[email protected]> Signed-off-by: Jan Hentschel <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23653 Expose content of meta table in web ui; addendum (#1061) Fix error prone problem Signed-off-by: Nick Dimiduk <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> Signed-off-by: stack <[email protected]> * HBASE-23680 RegionProcedureStore missing cleaning of hfile archive (#1022) Signed-off-by: stack <[email protected]> * HBASE-23661 Reduced number of Checkstyle violations in hbase-rest Signed-off-by: Peter Somogyi <[email protected]> Signed-off-by: Sean Busbey <[email protected]> * HBASE-23686 Revert binary incompatible change in ByteRangeUtils and removed reflections in CommonFSUtils Signed-off-by: Sean Busbey <[email protected]> * HBASE-23156 start-hbase.sh failed with ClassNotFoundException when build with hadoop3 (#1067) Signed-off-by: Duo Zhang <[email protected]> * HBASE-23347 Allow custom authentication methods for RPCs; addendum (#1060) Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23055 Alter hbase:meta (#1043) Make hbase:meta region schema dynamic. Patch has been under development a good while and its focus has changed a few times so its bloated with fixup from older versions. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java Shut down access to internals and removed unused methods. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java Cleanup/refactor section on replica-handling. M hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java Get hbase:meta schema from filesystem rather than from hard-coding. * HBASE-20516 Offheap read-path needs more detail (#1081) Signed-off-by: Anoop Sam John <[email protected]> * HBASE-23711 - Add test for MinVersions and KeepDeletedCells TTL (#1079) Signed-off-by: stack <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23709 Unwrap the real user to properly dispatch proxy-user auth'n REST and Thrift servers started failing because the check in BuiltinProviderSelector wasn't checking the "real" user for kerberos credentials. This resulted in the KerberosAuthnProvider not being invoked when it should have been. Closes #1080 Signed-off-by: Peter Somogyi <[email protected]> * HBASE-23719 Add 1.5.0 release to Downloads (#1083) Signed-off-by: Jan Hentschel <[email protected]> * HBASE-23720 [create-release] Update yetus version used from 0.11.0 to 0.11.1 * HBASE-23705 Add CellComparator to HFileContext (#1062) Codecs don't have access to what CellComparator to use. Backfill. M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java Adds a new compareRows with default implementation that takes a ByteBuffer. Needed by the index in a block encoder implementation. M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java Adds implementation for meta of new compareRows method. Adds utility method for figuring comparator based off tablename. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java Comparator is in context. Remove redundant handling. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java Comparator is in context. Remove redundant handling. Clean javadoc. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java Clean javadoc. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java Cache context so can use it to get comparator to use later. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java Cache cellcomparator to use. Javdoc on diff between HFileContext and HFileInfo. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java Add CellComparator M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java Remove comparator caching. Get from context instead. M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java Skip a reflection if we can. M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java Javadoc. Removed unused filed. Signed-off-by: Anoop Sam John <[email protected]> Signed-off-by: Ramkrishna <[email protected]> Signed-off-by: Jan Hentschel <[email protected]> * HBASE-23715 MasterFileSystem should not create MasterProcWALs dir on … (#1078) Signed-off-by: Josh Elser <[email protected]> * HBASE-23069 periodic dependency bump for Sep 2019 (#1082) Signed-off-by: Nick Dimiduk <[email protected]> * HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (fix bloomfilters… (#1012) Set encoding and blooms on meta as default. Also shutdown access to the initial meta schema creating method; get from TableDescriptors if you need access to schema or edit it as you would any other table if you want to edit it. Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23069 periodic dependency bump for Sep 2019 (#1082); ADDENDUM For asciidoctor, s/1.5.8/1.5.8.1/. * HBASE-23722 Real user might be null in non-proxy-user case Closes #1085 Signed-off-by: stack <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23069 periodic dependency bump for Sep 2019 (#1082); ADDENDUM Remove staging repo added by mistake. * HBASE-23069 periodic dependency bump for Sep 2019 (#1082); ADDENDUM AND.... undo thirdparty testing version update. * HBASE-23710 - Priority configuration for system coprocessors (#1077) Signed-off-by: Viraj Jasani <[email protected]> * HBASE-23729 [Flakeys] TestRSGroupsBasics#testClearNotProcessedDeadServer fails most of the time * HBASE-23733 [Flakey Tests] TestSplitTransactionOnCluster * HBASE-23726 Forward-port HBASE-21345 to branch-2.2, 2.3 & master as well. HBASE-21345 - [hbck2] Allow version check to proceed even though master is 'initializing'. Just remove the check state from the getClusterStatus call. Signed-off-by: Michael Stack <[email protected]> Signed-off-by: Peter Somogyi <[email protected]> Signed-off-by: Sakthi <[email protected]> (cherry picked from commit dd8496a5460693c49ec0bf5475ef79e40457e6bd) * HBASE-23735 [Flakey Tests] TestClusterRestartFailover & TestClusterRestartFailoverSplitWithoutZk * HBASE-23737 [Flakey Tests] TestFavoredNodeTableImport fails 30% of the time; AMENDMENT This is actual fix; previous added debug to test. * HBASE-23737 [Flakey Tests] TestFavoredNodeTableImport fails 30% of the time Second part of this issue -- changes to the test. Co-authored-by: Wellington Ramos Chevreuil <[email protected]> Co-authored-by: meiyi <[email protected]> Co-authored-by: Sakthi <[email protected]> Co-authored-by: Viraj Jasani <[email protected]> Co-authored-by: Ankit Singhal <[email protected]> Co-authored-by: Andrew Purtell <[email protected]> Co-authored-by: Pierre Zemb <[email protected]> Co-authored-by: Karthik Palanisamy <[email protected]> Co-authored-by: Sean Busbey <[email protected]> Co-authored-by: Duo Zhang <[email protected]> Co-authored-by: Peter Somogyi <[email protected]> Co-authored-by: binlijin <[email protected]> Co-authored-by: Wei-Chiu Chuang <[email protected]> Co-authored-by: Pankaj <[email protected]> Co-authored-by: Michael Stack <[email protected]> Co-authored-by: chenxu14 <[email protected]> Co-authored-by: ravowlga123 <[email protected]> Co-authored-by: BukrosSzabolcs <[email protected]> Co-authored-by: Shardul Singh <[email protected]> Co-authored-by: Nick Dimiduk <[email protected]> Co-authored-by: Bharath Vissapragada <[email protected]> Co-authored-by: Dice <[email protected]> Co-authored-by: Geoffrey Jacoby <[email protected]> Co-authored-by: Mingliang Liu <[email protected]> Co-authored-by: Reid Chan <[email protected]> Co-authored-by: belugabehr <[email protected]> Co-authored-by: Udai Bhan Kashyap <[email protected]> Co-authored-by: Baiqiang Zhao <[email protected]> Co-authored-by: bsglz <[email protected]> Co-authored-by: xuqinya1 <[email protected]> Co-authored-by: Kevin Risden <[email protected]> Co-authored-by: Guangxu Cheng <[email protected]> Co-authored-by: Sandeep Pal <[email protected]> Co-authored-by: Guanghao Zhang <[email protected]> Co-authored-by: XinSun <[email protected]> Co-authored-by: Toshihiro Suzuki <[email protected]> Co-authored-by: Junegunn Choi <[email protected]> Co-authored-by: lhofhansl <[email protected]> Co-authored-by: Andor Molnár <[email protected]> Co-authored-by: Beata Sudi <[email protected]> Co-authored-by: ramkrish86 <[email protected]> Co-authored-by: Jan Hentschel <[email protected]> Co-authored-by: Manu Manjunath <[email protected]> Co-authored-by: Yiran Wu <[email protected]> Co-authored-by: Aman Poonia <[email protected]> Co-authored-by: Josh Elser <[email protected]> Co-authored-by: WenFeiYi <[email protected]> Co-authored-by: bzhaoopenstack <[email protected]>
asfgit
pushed a commit
that referenced
this pull request
Jan 27, 2020
Revert of a revert, i.e., putting this patch back. This reverts commit f4b0ad9.
thangTang
pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Codecs don't have access to what CellComparator to use. Backfill. M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java Adds a new compareRows with default implementation that takes a ByteBuffer. Needed by the index in a block encoder implementation. M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java Adds implementation for meta of new compareRows method. Adds utility method for figuring comparator based off tablename. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java Comparator is in context. Remove redundant handling. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java Comparator is in context. Remove redundant handling. Clean javadoc. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java Clean javadoc. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java Cache context so can use it to get comparator to use later. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java Cache cellcomparator to use. Javdoc on diff between HFileContext and HFileInfo. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java Add CellComparator M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java Remove comparator caching. Get from context instead. M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java Skip a reflection if we can. M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java Javadoc. Removed unused filed. Signed-off-by: Anoop Sam John <[email protected]> Signed-off-by: Ramkrishna <[email protected]> Signed-off-by: Jan Hentschel <[email protected]>
thangTang
pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Codecs don't have access to what CellComparator to use. Backfill. M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java Adds a new compareRows with default implementation that takes a ByteBuffer. Needed by the index in a block encoder implementation. M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java Adds implementation for meta of new compareRows method. Adds utility method for figuring comparator based off tablename. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java Comparator is in context. Remove redundant handling. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java Comparator is in context. Remove redundant handling. Clean javadoc. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java Clean javadoc. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java Cache context so can use it to get comparator to use later. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java Cache cellcomparator to use. Javdoc on diff between HFileContext and HFileInfo. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java Add CellComparator M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java Remove comparator caching. Get from context instead. M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java Skip a reflection if we can. M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java Javadoc. Removed unused filed. Signed-off-by: Anoop Sam John <[email protected]> Signed-off-by: Ramkrishna <[email protected]> Signed-off-by: Jan Hentschel <[email protected]>
infraio
pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
Codecs don't have access to what CellComparator to use. Backfill. M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java Adds a new compareRows with default implementation that takes a ByteBuffer. Needed by the index in a block encoder implementation. M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java Adds implementation for meta of new compareRows method. Adds utility method for figuring comparator based off tablename. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java Comparator is in context. Remove redundant handling. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java Comparator is in context. Remove redundant handling. Clean javadoc. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java Clean javadoc. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java Cache context so can use it to get comparator to use later. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java Cache cellcomparator to use. Javdoc on diff between HFileContext and HFileInfo. M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java Add CellComparator M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java Remove comparator caching. Get from context instead. M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java Skip a reflection if we can. M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java Javadoc. Removed unused filed. Signed-off-by: Anoop Sam John <[email protected]> Signed-off-by: Ramkrishna <[email protected]> Signed-off-by: Jan Hentschel <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Codecs don't have access to what CellComparator to use. Backfill.
M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java
Adds a new compareRows with default implementation that takes a ByteBuffer.
Needed by the index in a block encoder implementation.
M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java
Adds implementation for meta of new compareRows method. Adds utility
method for figuring comparator based off tablename.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java
Comparator is in context. Remove redundant handling.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
Comparator is in context. Remove redundant handling. Clean javadoc.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java
Clean javadoc.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java
Cache context so can use it to get comparator to use later.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
Cache cellcomparator to use. Javdoc on diff between HFileContext and
HFileInfo.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java
Add CellComparator
M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java
Remove comparator caching. Get from context instead.
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
Skip a reflection if we can.
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java
Javadoc. Removed unused filed.