Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-26208 Supports revoke @ns single permission #3626

Closed
wants to merge 4,814 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
4814 commits
Select commit Hold shift + click to select a range
081ea2d
HBASE-26747 Use python2 instead of python in our python scripts (#4102)
Apache9 Feb 9, 2022
2dd3d60
HBASE-26697 Migrate HBase Nightly HBase-Flaky-Tests and HBase-Find-Fl…
Apache9 Jan 22, 2022
0bc8671
HBASE-26747 Addendum remove python from Jenkinsfile and Jenkinsfile_G…
Apache9 Feb 9, 2022
cbbb1a0
HBASE-26724 Backport the UT changes in HBASE-24510 to branch-2.x (#4081)
Apache9 Feb 9, 2022
d747da6
HBASE-26709 Ban the usage of junit 3 TestCase (#4065)
Apache9 Jan 27, 2022
a890fad
HBASE-26741 Incorrect exception handling in shell (#4101)
petersomogyi Feb 10, 2022
d242c8f
HBASE-26521 Name RPC spans as `$package.$service/$method` (#4024)
ndimiduk Feb 9, 2022
83f890f
HBASE-26747 Addendum use python2 directly in Jenkinsfile and Jenkinsf…
Apache9 Feb 10, 2022
a2e64ff
HBASE-26749 Migrate HBase main pre commit job to ci-hbase (#4107)
Apache9 Feb 11, 2022
1bee604
HBASE-26688 Threads shared EMPTY_RESULT may lead to unexpected client…
YutSean Feb 15, 2022
55a83ac
HBASE-26434 Do compact when all L0 files are expired (#3830)
sunhelly Feb 15, 2022
3e84e50
HBASE-26730 Extend hbase shell 'status' command to support an option …
apurtell Feb 16, 2022
6906291
HBASE-26742 Comparator of NOT_EQUAL NULL is invalid for checkAndMutat…
sunhelly Feb 16, 2022
07d6381
HBASE-26712 Balancer encounters NPE in rare case (#4112) (#4092)
comnetwork Feb 16, 2022
cb225f1
HBASE-26659 The ByteBuffer of metadata in RAMQueueEntry in BucketCach…
YutSean Feb 18, 2022
1145581
HBASE-26576 Allow pluggable queue to be used with the fast path execu…
rmarsch Feb 18, 2022
2a7b413
HBASE-26727 Fix CallDroppedException reporting (#4088)
bbeaudreault Feb 19, 2022
5118e71
HBASE-26703 Allow configuration of IPC queue balancer (#4063)
bbeaudreault Feb 19, 2022
b265fe5
HBASE-26673 Implement a shell command for change SFT implementation (…
2005hithlj Feb 19, 2022
b9b7a65
HBASE-26546 hbase-shaded-client missing required thirdparty classes u…
bbeaudreault Feb 22, 2022
e1131a2
HBASE-26767 Rest server should not use a large Header Cache.
ss77892 Feb 23, 2022
4cdb380
HBASE-26640 Reimplement master local region initialization to better …
Apache9 Feb 24, 2022
82282a9
HBASE-26765 Minor refactor of async scanning code (#4121)
ndimiduk Feb 24, 2022
77bb153
HBASE-26707: Reduce number of renames during bulkload (#4066) (#4122)
BukrosSzabolcs Feb 25, 2022
b3292b8
HBASE-26663 Upgrade Maven Enforcer Plugin
ndimiduk Feb 23, 2022
4ad66df
HBASE-26772 Shell suspended in background (#4137)
petersomogyi Mar 1, 2022
7f30aa8
HBASE-23303 Add default security headers if SSL is enabled (#4128)
anmolnar Mar 2, 2022
e93abe6
HBASE-26776 RpcServer failure to SASL handshake always logs user "unk…
ndimiduk Mar 1, 2022
c5f6355
HBASE-26762 Un-Deprecate and improve documentation for Scan#setRowPre…
nielsbasjes Mar 2, 2022
d020cba
HBASE-26745: Include balancer cost metrics in jmx endpoint (#4140)
briaugenreich Mar 3, 2022
8a9e083
HBASE-26622 Update error-prone to 2.10
madrob Dec 23, 2021
651ec82
HBASE-26788 Disable Checks API callback from test results in PRs
ndimiduk Mar 2, 2022
87ce575
HBASE-26764 Implement generic exception support for TraceUtil methods…
ndimiduk Mar 4, 2022
2fa4bb0
HBASE-26242 Allow split when store file count larger than the confige…
sunhelly Mar 4, 2022
b5ac638
Revert "HBASE-26242 Allow split when store file count larger than the…
sunhelly Mar 4, 2022
df90a4b
HBASE-26242 Allow split when store file count larger than the confige…
sunhelly Mar 4, 2022
4644efb
HBASE-26691 Replacing log4j with reload4j for branch-2.x (#4050)
Apache9 Mar 4, 2022
bed30b1
Amend HBASE-26691 Replacing log4j with reload4j for branch-2.x
apurtell Mar 4, 2022
eb1c57d
HBASE-26777 BufferedDataBlockEncoder$OffheapDecodedExtendedCell.deepC…
stoty Mar 4, 2022
fc92a00
HBASE-26731 Add metrics for active and expired scanners (#4145)
bbeaudreault Mar 4, 2022
961796a
[branch-2] Amend HBASE-26731 Add metrics for active and expired scanners
apurtell Mar 4, 2022
01acd87
HBASE-26759 Fix trace continuity through CallRunner (#4126)
ndimiduk Mar 4, 2022
0a7e49c
HBASE-26792 Implement ScanInfo#toString (#4154)
shahrs87 Mar 5, 2022
1aff89e
HBASE-25709 Close region may stuck when region is compacting and skip…
sunhelly Mar 7, 2022
f4c91ee
HBASE-26552 Introduce retry to logroller to avoid abort (#4170)
sunhelly Mar 7, 2022
2e2764f
HBASE-26760 LICENSE handling should not allow non-aggregated apache-2…
busbey Mar 7, 2022
5bae04e
HBASE-26784 Use HIGH_QOS for ResultScanner.close requests (#4163)
bbeaudreault Mar 8, 2022
98836fb
HBASE-26784 Addendum: Close scanner request should properly inherit o…
bbeaudreault Mar 8, 2022
300f9b9
HBASE-26582 Prune use of Random and SecureRandom objects (#4118)
apurtell Mar 8, 2022
7e267c6
HBASE-26782 Minor code cleanup in and around RpcExecutor
ndimiduk Mar 1, 2022
91cb302
HBASE-26783 ScannerCallable doubly clears meta cache on retries (#4147)
bbeaudreault Mar 9, 2022
8a43178
HBASE-26814 Default StoreHotnessProtector to off, with logs to guide …
bbeaudreault Mar 9, 2022
e9065e7
HBASE-26815 TestFanOutOneBlockAsyncDFSOutput is flakey (#4186)
xicm Mar 10, 2022
3ebb8c1
HBASE-26804 Missing opentelemetry agent in hadoop-two-compat.xml (#4185)
taklwu Mar 10, 2022
fac011e
HBASE-26814 Addendum: Fix spotbugs warning and clarify intent (#4201)
bbeaudreault Mar 10, 2022
49d3a00
HBASE-26715 Blocked on SyncFuture in AsyncProtobufLogWriter#write (#4…
apurtell Mar 10, 2022
8c42bca
HBASE-26816 Fix CME in ReplicationSourceManager (#4187)
sunhelly Mar 11, 2022
0b879f2
HBASE-26824 TestHBaseTestingUtil.testResolvePortConflict failing afte…
apurtell Mar 11, 2022
e46fbed
HBASE-26761 TestMobStoreScanner (testGetMassive) can OOME (#4204)
apurtell Mar 11, 2022
51fd863
HBASE-26817 Mark RpcExecutor as IA.LimitedPrivate COPROC and PHOENIX
ndimiduk Mar 9, 2022
a227d76
HBASE-26819 Minor code cleanup in and around RpcScheduler
ndimiduk Mar 9, 2022
340cc6c
HBASE-26802 Backport the log4j2 changes to branch-2 (#4166)
Apache9 Mar 11, 2022
9e1de13
HBASE-25465 Use javac --release option for supporting cross version c…
Apache9 Mar 12, 2022
0f93663
HBASE-26323 Introduce a Snapshot Procedure (#4115)
frostruan Mar 12, 2022
05627b7
HBASE-26830 Rewrite TestLruBlockCache to make it more stable (#4212)
Apache9 Mar 14, 2022
2a5b340
HBASE-26828 Increase the concurrency when running UTs in pre commit j…
Apache9 Mar 14, 2022
50e1230
HBASE-26670 HFileLinkCleaner should be added even if snapshot is disa…
mymeiyi Mar 14, 2022
cdabd6a
HBASE-25895 Implement a Cluster Metrics JSON endpoint
ndimiduk May 15, 2021
e6879b8
HBASE-25895 Implement a Cluster Metrics JSON endpoint (addendum)
ndimiduk Mar 11, 2022
350db51
HBASE-26835 Rewrite TestLruAdaptiveBlockCache to make it more stable …
Apache9 Mar 15, 2022
a4b192e
HBASE-26611 Changing SFT implementation on disabled table is dangerou…
Apache9 Mar 15, 2022
46af73b
HBASE-26840 Fix NPE in the retry of logroller (#4224)
sunhelly Mar 15, 2022
c1bf2ff
HBASE-26833 Avoid waiting to clear buffer usage of ReplicationSourceS…
sunhelly Mar 15, 2022
3c968a1
HBASE-25896 Implement a Region Visualization on Master WebUI (#4228)
ndimiduk Mar 15, 2022
4dff67c
HBASE-26848 Set java.io.tmpdir on mvn command when running jenkins jo…
Apache9 Mar 16, 2022
d3629bb
HBASE-26837 Set SFT config when creating TableDescriptor in TestClone…
Apache9 Mar 16, 2022
db5e954
HBASE-26680 Close and do not write trailer for the broken WAL writer …
sunhelly Mar 16, 2022
50757ef
Hbase 26275 update error message when executing deleteall with ROWPRE…
xicm Mar 16, 2022
fdc7c49
HBASE-26720 ExportSnapshot should validate the source snapshot before…
xicm Mar 16, 2022
26af3b9
HBASE-26828 Addendum fix syntax error
Apache9 Mar 16, 2022
abde344
HBASE-26813 Remove javax.ws.rs-api dependency (#4191)
ndimiduk Mar 16, 2022
b5d5fe8
HBASE-26848 Addendum make tmpdir absolute
Apache9 Mar 17, 2022
81398f9
HBASE-26833 Addendum fix NPE in TestReplicationSource.testTerminateCl…
Apache9 Mar 17, 2022
4a6ba11
HBASE-26842 TestSnapshotProcedure fails in branch-2 (#4225)
Mar 18, 2022
ac5651c
HBASE-26858 Refactor TestMasterRegionOnTwoFileSystems to avoid dead l…
Apache9 Mar 18, 2022
7af08ce
HBASE-26861 Fix flaky TestSnapshotFromMaster.testSnapshotHFileArchivi…
Apache9 Mar 18, 2022
d09b04d
HBASE-26855 Delete unnecessary dependency on jaxb-runtime jar (#4236)
ndimiduk Mar 18, 2022
ba14796
Revert "HBASE-26813 Remove javax.ws.rs-api dependency (#4191)"
Apache9 Mar 19, 2022
8640453
HBASE-26870 Log4j2 integration is incorrect in nighly's client integr…
Apache9 Mar 21, 2022
321c35a
HBASE-26834 Adapt ConnectionRule for both sync and async connections
ndimiduk Mar 21, 2022
0d968af
HBASE-26531 Trace coprocessor exec endpoints
ndimiduk Feb 16, 2022
0a30289
HBASE-26531 Trace coprocessor exec endpoints (addendum)
ndimiduk Mar 14, 2022
68f3fbe
HBASE-26791 Memstore flush fencing issue for SFT (#4202)
Apache9 Mar 21, 2022
d889746
HBASE-26869 RSRpcServices.scan should deep clone cells when RpcCallCo…
comnetwork Mar 23, 2022
c55c289
Revert "HBASE-26869 RSRpcServices.scan should deep clone cells when R…
comnetwork Mar 23, 2022
5affa20
HBASE-26869 RSRpcServices.scan should deep clone cells when RpcCallCo…
comnetwork Mar 23, 2022
1caa288
HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (#4268)
bbeaudreault Mar 23, 2022
b7cfc1d
HBASE-26175 MetricsHBaseServer should record all kinds of Exceptions …
sunhelly Mar 24, 2022
ef40c04
HBASE-26878 TableInputFormatBase should cache RegionSizeCalculator (#…
bbeaudreault Mar 24, 2022
6650807
HBASE-26881 Backport HBASE-25368 to branch-2 (#4267)
wchevreuil Mar 25, 2022
51f8c87
HBASE-26875 RpcRetryingCallerImpl translateException ignores return v…
bbeaudreault Mar 25, 2022
c73ef2c
HBASE-26876 Use toStringBinary for rowkey in RegionServerCallable err…
bbeaudreault Mar 25, 2022
bd5b447
HBASE-26871 shaded mapreduce and shaded byo-hadoop client artifacts c…
busbey Mar 26, 2022
9ecd4e7
HBASE-26864 SplitTableRegionProcedure calls openParentRegions() at a …
huaxiangsun Mar 26, 2022
7ae6be7
HBASE-26872 Load rate calculator for cost functions should be more pr…
sunhelly Mar 27, 2022
fd621a7
HBASE-26832 Avoid repeated releasing of flushed wal entries in AsyncF…
Apache9 Mar 27, 2022
7623900
Revert "HBASE-26872 Load rate calculator for cost functions should be…
sunhelly Mar 28, 2022
e0ae456
HBASE-26617 Use spotless to reduce the pain on fixing checkstyle issu…
Apache9 Mar 28, 2022
89e7050
HBASE-26617 Addendum add removeUnusedImports
Apache9 Mar 28, 2022
170570b
HBASE-26872 Load rate calculator for cost functions should be more pr…
sunhelly Mar 27, 2022
0dbcf8f
HBASE-26718 HFileArchiver can remove referenced StoreFiles from the a…
d-c-manning Mar 28, 2022
5308eb3
HBASE-26896 list_quota_snapshots fails with ‘ERROR NameError: uniniti…
stoty Mar 28, 2022
af5624b
HBASE-26885 The TRSP should not go on when it get a bogus server name…
bsglz Mar 29, 2022
cd09f3a
HBASE-26902 Bump bcprov-jdk15on from 1.60 to 1.67 (#4292)
dependabot[bot] Mar 29, 2022
bb18f40
HBASE-26871 addendum. use the jar command from JAVA_HOME (#4297)
busbey Mar 30, 2022
455b7c7
HBASE-26903 Bump httpclient from 4.5.3 to 4.5.13 (#4296)
apurtell Mar 30, 2022
1a85588
HBASE-26812 ShortCircuitingClusterConnection fails to close RegionSca…
comnetwork Mar 31, 2022
98db15a
HBASE-26245 Store region server list in master local region (#4136)
Apache9 Mar 31, 2022
489af42
HBASE-26545 Implement tracing of scan
ndimiduk Mar 31, 2022
35868ab
HBASE-26838 Junit jar is not included in the hbase tar ball, causing …
wchevreuil Mar 31, 2022
9ea77ef
HBASE-26912 Bump checkstyle from 8.28 to 8.29 (#4293)
dependabot[bot] Mar 31, 2022
4311974
HBASE-26811 Secondary replica may be disabled for read incorrectly fo…
comnetwork Apr 1, 2022
03b5529
HBASE-26916 Fix missing braces warnings in DefaultVisibilityExpressio…
Apache9 Apr 3, 2022
0b87621
HBASE-26919 Rewrite the counting rows part in TestFromClientSide4 (#4…
Apache9 Apr 3, 2022
1560b41
HBASE-26920 Fix missing braces warnings in TestProcedureMember (#4315)
Apache9 Apr 3, 2022
ab9ccab
HBASE-26921 Rewrite the counting cells part in TestMultiVersions (#4316)
Apache9 Apr 3, 2022
3b38380
HBASE-26924 Fix log parameter error and spelling error (#4318)
weimingdiit Apr 4, 2022
8ef1ae3
HBASE-26885 Addendum throw exception instead of return in TRSP to let…
bsglz Apr 4, 2022
1c14046
HBASE-26927 Add snapshot scanner UT with SFT and some cleanups to Tes…
wchevreuil Apr 5, 2022
8755509
HBASE-26882 Backport "HBASE-26810 Add dynamic configuration support f…
taklwu Apr 5, 2022
197b8d4
HBASE-26928 Fix several indentation problems (#4323)
Apache9 Apr 6, 2022
28acbc1
HBASE-26929 Upgrade surefire plugin to 3.0.0-M6 (#4319)
Apache9 Apr 7, 2022
eb22319
HBASE-26939 Typo in admin.rb "COMPRESSION_COMPACT_MAJPR" (#4331)
apurtell Apr 8, 2022
e7eb628
HBASE-26922 Fix LineLength warnings as much as possible if it can not…
Apache9 Apr 9, 2022
d001fb0
HBASE-26880 Misspelling commands in hbase shell will crash the shell …
petersomogyi Apr 11, 2022
f9abe3d
HBASE-26618 Involving primary meta region in meta scan with CatalogRe…
huaxiangsun Apr 11, 2022
52ca440
HBASE-26901 delete with null columnQualifier occurs NullPointerExcept…
eomiks Apr 12, 2022
5dc93d7
HBASE-26895 on hbase shell, 'delete/deleteall' for a columnfamily is …
eomiks Apr 12, 2022
f8b2ac0
HBASE-26941 LocalHBaseCluster.waitOnRegionServer should quit while th…
Apache9 Apr 13, 2022
b405a6f
Revert "HBASE-26941 LocalHBaseCluster.waitOnRegionServer should quit …
Apache9 Apr 13, 2022
c76daa1
HBASE-26944 Possible resource leak while creating new region scanner …
pankaj72981 Apr 15, 2022
bbe048c
HBASE-26946 TestRetainAssignmentOnRestartSplitWithoutZk infinite loop…
Apache9 Apr 16, 2022
ae8eccd
HBASE-26938 Compaction failures after StoreFileTracker integration (#…
Apache9 Apr 17, 2022
085c651
HBASE-26941 LocalHBaseCluster.waitOnRegionServer should not call join…
Apache9 Apr 17, 2022
a81b369
HBASE-26906 Remove duplicate dependency declaration
ndimiduk Mar 30, 2022
d7e6baf
HBASE-26947 Implement a special TestAppender to limit the size of tes…
Apache9 Apr 21, 2022
edbe44b
HBASE-25058 Export necessary modules when running under JDK11 (#4337)
Apache9 Apr 21, 2022
2240025
HBASE-26807 Unify CallQueueTooBigException special pause with CallDro…
bbeaudreault Apr 22, 2022
e5b0084
HBASE-26891 Make MetricsConnection scope configurable (#4365)
bbeaudreault Apr 22, 2022
8d17241
HBASE-26942 cache region locations when getAllRegionLocations (#4364)
frostruan Apr 22, 2022
0f9f6f2
HBASE-26959 Brotli compression support (#4353)
apurtell Apr 22, 2022
c9d833e
HBASE-26971 SnapshotInfo --snapshot param is marked as required even …
wchevreuil Apr 25, 2022
2fe5a4f
HBASE-26807 Addendum: adhere to maxAttempts in RawAsyncTableImpl.batc…
bbeaudreault Apr 25, 2022
c2baa1f
HBASE-26581 Add metrics for failed replication edits (#4347)
briaugenreich Apr 26, 2022
6813a5c
HBASE-26979 StoreFileListFile logs frequent stacktraces at INFO level…
apurtell Apr 28, 2022
ea5c5da
HBASE-26980 Update javadoc of BucketCache.java (#4374)
liangxs Apr 28, 2022
895e0f4
HBASE-26975 Add on heap and off heap memstore info in rs web UI (#4368)
liangxs Apr 28, 2022
fd2e720
HBASE-22349 slop in StochasticLoadBalancer (#4371)
d-c-manning Apr 28, 2022
3090848
HBASE-26932 Skip generating ref guide when running 'mvn site' on bran…
Apache9 Apr 29, 2022
e9c9418
HBASE-26943 HMaster page style display confusion (#4369)
2005hithlj Apr 25, 2022
0a0bea6
HBASE-26917 Do not add --threads when running 'mvn site' (#4354)
Apache9 Apr 29, 2022
afb8afd
HBASE-26988 dynamic configuration of loadbalance.bytable (#4384)
d-c-manning Apr 29, 2022
191e932
HBASE-26992 Brotli compressor has unexpected behavior during reinitia…
apurtell Apr 30, 2022
731da45
HBASE-26899 tweak pom and formatter
Apache9 Apr 26, 2022
1a5b1b2
HBASE-26899 Run spotless:apply
Apache9 May 1, 2022
be8a9a5
HBASE-26989 TestStochasticLoadBalancer fixes for performance and cons…
d-c-manning May 2, 2022
a38716c
HBASE-26648 Improve fidelity of RegionLocator spans
ndimiduk Feb 11, 2022
1b5c881
HBASE-26648 Improve fidelity of RegionLocator spans (addendum)
ndimiduk May 2, 2022
3bafbf1
HBASE-26892 Add spotless:check in our pre commit general check (#4393)
Apache9 May 3, 2022
a025844
HBASE-26990 Add default implementation for BufferedMutator interface …
bbeaudreault May 3, 2022
9a41117
HBASE-26995 Remove ref guide check in pre commit and nightly for bran…
Apache9 May 5, 2022
cb61d6a
HBASE-26977 HMaster's ShutdownHook does not take effect, if tablesOnM…
2005hithlj May 3, 2022
8703cac
HBASE-26984 Chaos Monkey thread dies in ITBLL Chaos GracefulRollingRe…
huaxiangsun May 5, 2022
91eab0b
HBASE-27000 Block cache stats (Misses Caching) display error in RS we…
tomscut May 6, 2022
f5a0806
HBASE-26963 ReplicationSource#removePeer hangs if we try to remove ba…
shahrs87 May 7, 2022
6cd31ed
HBASE-26330 Document new provided compression codecs (#4396)
apurtell May 7, 2022
ad2180b
HBASE-26994 MasterFileSystem create directory without permission chec…
skysiders May 8, 2022
7b7f576
HBASE-26999 HStore should try write WAL compaction marker before repl…
wchevreuil May 9, 2022
5cfce69
HBASE-27003 Optimize log format for PerformanceEvaluation (#4411)
tomscut May 11, 2022
b7647d8
HBASE-27021 StoreFileInfo should set its initialPath in a consistent …
wchevreuil May 12, 2022
e3359ef
Revert "HBASE-27021 StoreFileInfo should set its initialPath in a con…
Apache9 May 12, 2022
930bc53
HBASE-27021 StoreFileInfo should set its initialPath in a consistent way
wchevreuil May 10, 2022
62dbae2
HBASE-26999 Addendum fix spotless errors
Apache9 May 13, 2022
43e8c11
HBASE-27013 Introduce read all bytes when using pread for prefetch (#…
taklwu May 13, 2022
7bc7eb5
HBASE-26905 ReplicationPeerManager#checkPeerExists should throw Repli…
shahrs87 May 14, 2022
2efaa7d
HBASE-27019 Minor compression performance improvements (#4420)
apurtell May 14, 2022
31f4b42
HBASE-27032 The draining region servers metric description is incorre…
tomscut May 17, 2022
de9a17e
HBASE-26856:BufferedDataBlockEncoder.OnheapDecodedCell value can get …
arshadmohammad May 17, 2022
71bd62d
Revert "HBASE-26856:BufferedDataBlockEncoder.OnheapDecodedCell value …
ndimiduk May 18, 2022
1e6d93a
HBASE-26986 Trace a one-shot execution of a Master procedure
ndimiduk Apr 28, 2022
b9b5302
HBASE-27045 Disable TestClusterScopeQuotaThrottle (#4440)
Apache9 May 18, 2022
f8e742d
HBASE-27024 The User API and Developer API links are broken on hbase.…
Apache9 May 13, 2022
f20f771
HBASE-27006 Move nightly integration testing to new larger test node …
busbey May 18, 2022
8a3a942
HBASE-27018 Add a tool command list_liveservers (#4416)
tomscut May 19, 2022
a836fc8
HBASE-27050 Support unit test pattern matching again (#4447)
apurtell May 20, 2022
7af82cf
HBASE-27051 TestReplicationSource.testReplicationSourceInitializingMe…
apurtell May 20, 2022
2cae259
HBASE-27052 TestAsyncTableScanner.testScanWrongColumnFamily is flaky …
ndimiduk May 20, 2022
8bc55f8
HBASE-27043 Let lock wait timeout to improve performance of SnapshotH…
sunhelly May 20, 2022
b00e303
HBASE-27054 TestStochasticLoadBalancerRegionReplicaLargeCluster.testR…
d-c-manning May 21, 2022
a6b1c7e
HBASE-27027 Use jetty SslContextFactory.Server instead of deprecated …
bitterfox May 22, 2022
bc40ede
HBASE-27047 Fix typo for metric drainingRegionServers (#4441)
tomscut May 22, 2022
d8d1089
HBASE-27030 Fix undefined local variable error in draining_servers.rb…
junegunn May 22, 2022
778ae2d
HBASE-26933 Remove all ref guide stuff on branch other than master (#…
Apache9 May 22, 2022
4260e3a
HBASE-26933 Addendum remove unused resources and links on site
Apache9 May 22, 2022
200cbcc
HBASE-26320 Implement a separate thread pool for the LogCleaner (#371…
sunhelly May 24, 2022
56b81b4
HBASE-26342 Support custom paths of independent configuration and poo…
sunhelly May 24, 2022
02c1fef
HBASE-27055 Add additional comments when using HBASE_TRACE_OPTS with …
taklwu May 24, 2022
50dd961
HBASE-27061 two phase bulkload is broken when SFT is in use. (#4465)
ss77892 May 26, 2022
20895c7
HBASE-26854 Shell startup logs a bunch of noise (#4469)
apurtell May 27, 2022
779e985
HBASE-27065 [branch-2] Build against Hadoop 3.3.3 (#4471)
apurtell May 27, 2022
5e54c6d
HBASE-27066 The Region Visualizer display failed (#4472)
tomscut May 29, 2022
7fbfb10
HBASE-27069 Hbase SecureBulkload permission regression (#4475)
stoty May 31, 2022
cfba2f2
HBASE-27064 CME in TestRegionNormalizerWorkQueue (#4468)
apurtell May 31, 2022
f4f3e63
HBASE-27068 NPE occurs when the active master has not yet been electe…
tomscut May 31, 2022
59700ba
HBASE-27079 Lower some DEBUG level logs in ReplicationSourceWALReader…
apurtell May 31, 2022
e69da68
HBASE-26063 The current checkcompatibility.py script can not compare …
Apache9 Jul 3, 2021
6dd1a6b
HBASE-27081 Fix disallowed compatibility breaks on branch-2.5 and bra…
apurtell Jun 1, 2022
716a56c
HBASE-27082 Change the return value of RSGroupInfo.getServers from So…
Apache9 Jun 1, 2022
bcf3cd5
Amend HBASE-27082 spotless:apply
apurtell Jun 1, 2022
40c2743
HBASE-27023 Fix license issues after running spotless:apply (#4458)
Apache9 Jun 2, 2022
b21930f
HBASE-27037 Display the region server state on the Web UI (#4432)
tomscut Jun 2, 2022
8270b5d
HBASE-27039 Some methods of MasterRegion should be annotated for test…
2005hithlj Jun 2, 2022
7a02cbf
HBASE-26962: Add mob info in web UI (#4359)
liangxs Jun 2, 2022
f61ac22
HBASE-26680 Close and do not write trailer for the broken WAL writer(…
sunhelly Jun 2, 2022
f97af48
HBASE-27046 The filenum in AbstractFSWAL should be monotone increasin…
Apache9 Jun 2, 2022
672b0a5
HBASE-26649 Support meta replica LoadBalance mode for RegionLocator#g…
huaxiangsun Jun 2, 2022
58aaed1
Revert "HBASE-26962: Add mob info in web UI (#4359)"
Jun 2, 2022
286c9c3
HBASE-26985 check permission for SecureBulkLoadManager (#4379)
skysiders Jun 2, 2022
bb8b80e
HBASE-26983 Upgrade JRuby to 9.3.4.0 (#4378)
busbey Jun 5, 2022
ef22184
HBASE-27017: MOB snapshot is broken when FileBased SFT is used (#4466)
BukrosSzabolcs Jun 5, 2022
fd6e6e0
HBASE-27080 Optimize debug output log of ConstantSizeRegionSplitPolic…
liuzhuang2017 Jun 6, 2022
eca7c60
HBASE-27084 Add spotless:check in mvn verify stage (#4482)
Apache9 Jun 6, 2022
50e2ef2
HBASE-27038 CellComparator should extend Serializable (#4492)
bbeaudreault Jun 6, 2022
a2dc48a
HBASE-27093 AsyncNonMetaRegionLocator:put Complete CompletableFuture …
xiaowangzhixiao Jun 7, 2022
63a35fa
HBASE-27092 Regionserver table on Master UI is broken (#4495)
tomscut Jun 8, 2022
33f633a
Revert "HBASE-27066 The Region Visualizer display failed (#4472)"
ndimiduk Jun 9, 2022
b3c9ef3
HBASE-27066 The Region Visualizer display failed
ndimiduk Jun 6, 2022
27f9182
HBASE-27103 All MR UTs are broken because of ClassNotFound (#4514)
apurtell Jun 9, 2022
1dcee76
HBASE-27095 HbckChore should produce a report
ndimiduk Jun 10, 2022
6a5fb76
HBASE-27102 Vacate the .idea folder in order to simplify spotless con…
ndimiduk Jun 9, 2022
9b8bb58
HBASE-26366 Provide meaningful parent spans to ZK interactions
ndimiduk Jun 10, 2022
57d3122
HBASE-26923 PerformanceEvaluation support encryption option (#4489)
wenwj0 Jun 11, 2022
b21377f
Revert "HBASE-27084 Add spotless:check in mvn verify stage (#4482)"
Apache9 Jun 11, 2022
18a1a4b
HBASE-27097 SimpleRpcServer is broken (#4521)
apurtell Jun 11, 2022
b8f0c61
Revert "HBASE-25709 Close region may stuck when region is compacting …
virajjasani Jun 12, 2022
96693e8
HBASE-26856 BufferedDataBlockEncoder.OnheapDecodedCell value can get …
arshadmohammad May 17, 2022
3b9491b
HBASE-26167 Allow users to not start zookeeper and dfs cluster when u…
xicm Jun 15, 2022
ef0ff27
HBASE-27117 Update the method comments for RegionServerAccounting (#4…
tomscut Jun 17, 2022
f7d2d12
HBASE-27125 The batch size of cleaning expired mob files should have …
sunhelly Jun 17, 2022
47e1f1e
HBASE-26208 Supports revoke @ns specified permission
Jun 18, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
HBASE-26330 Document new provided compression codecs (#4396)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
Signed-off-by: Viraj Jasani<virajjasani@apache.org
apurtell committed May 7, 2022
commit 6cd31edf5cecf00d250e4f8291c0d36307796016
256 changes: 222 additions & 34 deletions src/main/asciidoc/_chapters/compression.adoc
Original file line number Diff line number Diff line change
@@ -40,12 +40,14 @@ Compressors and data block encoding can be used together on the same ColumnFamil
.Changes Take Effect Upon Compaction
If you change compression or encoding for a ColumnFamily, the changes take effect during compaction.

Some codecs take advantage of capabilities built into Java, such as GZip compression. Others rely on native libraries. Native libraries may be available as part of Hadoop, such as LZ4. In this case, HBase only needs access to the appropriate shared library.

Other codecs, such as Google Snappy, need to be installed first.
Some codecs are licensed in ways that conflict with HBase's license and cannot be shipped as part of HBase.
Some codecs take advantage of capabilities built into Java, such as GZip compression.
Others rely on native libraries. Native libraries may be available via codec dependencies installed into
HBase's library directory, or, if you are utilizing Hadoop codecs, as part of Hadoop. Hadoop codecs
typically have a native code component so follow instructions for installing Hadoop native binary
support at <<hadoop.native.lib>>.

This section discusses common codecs that are used and tested with HBase.

No matter what codec you use, be sure to test that it is installed correctly and is available on all nodes in your cluster.
Extra operational steps may be necessary to be sure that codecs are available on newly-deployed nodes.
You can use the <<compression.test,compression.test>> utility to check that a given codec is correctly installed.
@@ -55,11 +57,69 @@ To enable a compressor for a ColumnFamily, see <<changing.compression,changing.c
To enable data block encoding for a ColumnFamily, see <<data.block.encoding.enable,data.block.encoding.enable>>.

.Block Compressors
* none
* Snappy
* LZO
* LZ4
* NONE
+
This compression type constant selects no compression, and is the default.
* BROTLI
+
https://en.wikipedia.org/wiki/Brotli[Brotli] is a generic-purpose lossless compression algorithm
that compresses data using a combination of a modern variant of the LZ77 algorithm, Huffman
coding, and 2nd order context modeling, with a compression ratio comparable to the best currently
available general-purpose compression methods. It is similar in speed with GZ but offers more
dense compression.
* BZIP2
+
https://en.wikipedia.org/wiki/Bzip2[Bzip2] compresses files using the Burrows-Wheeler block
sorting text compression algorithm and Huffman coding. Compression is generally considerably
better than that achieved by the dictionary- (LZ-) based compressors, but both compression and
decompression can be slow in comparison to other options.
* GZ
+
gzip is based on the https://en.wikipedia.org/wiki/Deflate[DEFLATE] algorithm, which is a
combination of LZ77 and Huffman coding. It is universally available in the Java Runtime
Environment so is a good lowest common denominator option. However in comparison to more modern
algorithms like Zstandard it is quite slow.
* LZ4
+
https://en.wikipedia.org/wiki/LZ4_(compression_algorithm)[LZ4] is a lossless data compression
algorithm that is focused on compression and decompression speed. It belongs to the LZ77 family
of compression algorithms, like Brotli, DEFLATE, Zstandard, and others. In our microbenchmarks
LZ4 is the fastest option for both compression and decompression in that family, and is our
universally recommended option.
* LZMA
+
https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Markov_chain_algorithm[LZMA] is a
dictionary compression scheme somewhat similar to the LZ77 algorithm that achieves very high
compression ratios with a computationally expensive predictive model and variable size
compression dictionary, while still maintaining decompression speed similar to other commonly used
compression algorithms. LZMA is superior to all other options in general compression ratio but as
a compressor it can be extremely slow, especially when configured to operate at higher levels of
compression.
* LZO
+
https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Oberhumer[LZO] is another LZ-variant
data compression algorithm, with an implementation focused on decompression speed. It is almost
but not quite as fast as LZ4.
* SNAPPY
+
https://en.wikipedia.org/wiki/Snappy_(compression)[Snappy] is based on ideas from LZ77 but is
optimized for very high compression speed, achieving only a "reasonable" compression in trade.
It is as fast as LZ4 but does not compress quite as well. We offer a pure Java Snappy codec
that can be used instead of GZ as the universally available option for any Java runtime on any
hardware architecture.
* ZSTD
+
https://en.wikipedia.org/wiki/Zstd[Zstandard] combines a dictionary-matching stage (LZ77) with
a large search window and a fast entropy coding stage, using both Finite State Entropy and
Huffman coding. Compression speed can vary by a factor of 20 or more between the fastest and
slowest levels, while decompression is uniformly fast, varying by less than 20% between the
fastest and slowest levels.
+
ZStandard is the most flexible of the available compression codec options, offering a compression
ratio similar to LZ4 at level 1 (but with slightly less performance), compression ratios
comparable to DEFLATE at mid levels (but with better performance), and LZMA-alike dense
compression (and LZMA-alike compression speeds) at high levels; while providing universally fast
decompression.

.Data Block Encoding Types
Prefix::
@@ -122,16 +182,23 @@ The compression or codec type to use depends on the characteristics of your data

In general, you need to weigh your options between smaller size and faster compression/decompression. Following are some general guidelines, expanded from a discussion at link:https://lists.apache.org/thread.html/481e67a61163efaaf4345510447a9244871a8d428244868345a155ff%401378926618%40%3Cdev.hbase.apache.org%3E[Documenting Guidance on compression and codecs].

* In most cases, enabling LZ4 or Snappy by default is a good choice, because they have a low
performance overhead and provide reasonable space savings. A fast compression algorithm almost
always improves overall system performance by trading some increased CPU usage for better I/O
efficiency.
* If the values are large (and not pre-compressed, such as images), use a data block compressor.
* For [firstterm]_cold data_, which is accessed infrequently, depending on your use case, it might
make sense to opt for Zstandard at its higher compression levels, or LZMA, especially for high
entropy binary data, or Brotli for data similar in characteristics to web data. Bzip2 might also
be a reasonable option but Zstandard is very likely to offer superior decompression speed.
* For [firstterm]_hot data_, which is accessed frequently, you almost certainly want only LZ4,
Snappy, LZO, or Zstandard at a low compression level. These options will not provide as high of
a compression ratio but will in trade not unduly impact system performance.
* If you have long keys (compared to the values) or many columns, use a prefix encoder.
FAST_DIFF is recommended.
* If the values are large (and not precompressed, such as images), use a data block compressor.
* Use GZIP for [firstterm]_cold data_, which is accessed infrequently.
GZIP compression uses more CPU resources than Snappy or LZO, but provides a higher compression ratio.
* Use Snappy or LZO for [firstterm]_hot data_, which is accessed frequently.
Snappy and LZO use fewer CPU resources than GZIP, but do not provide as high of a compression ratio.
* In most cases, enabling Snappy or LZO by default is a good choice, because they have a low performance overhead and provide space savings.
* Before Snappy became available by Google in 2011, LZO was the default.
Snappy has similar qualities as LZO but has been shown to perform better.
* If enabling WAL value compression, consider LZ4 or SNAPPY compression, or Zstandard at
level 1. Reading and writing the WAL is performance critical. That said, the I/O
savings of these compression options can improve overall system performance.

[[hadoop.native.lib]]
=== Making use of Hadoop Native Libraries in HBase
@@ -235,11 +302,120 @@ Set in _hbase-env.sh_ the LD_LIBRARY_PATH environment variable when starting you
[[compressor.install]]
==== Configure HBase For Compressors

Before HBase can use a given compressor, its libraries need to be available.
Due to licensing issues, only GZ compression is available to HBase (via native Java libraries) in a default installation.
Other compression libraries are available via the shared library bundled with your hadoop.
The hadoop native library needs to be findable when HBase starts.
See
Compression codecs are provided either by HBase compressor modules or by Hadoop's native compression
support. As described above you choose a compression type in table or column family schema or in
site configuration using its short label, e.g. _snappy_ for Snappy, or _zstd_ for ZStandard. Which
codec implementation is dynamically loaded to support what label is configurable by way of site
configuration.

[options="header"]
|===
|Algorithm label|Codec implementation configuration key|Default value
//----------------------
|BROTLI|hbase.io.compress.brotli.codec|org.apache.hadoop.hbase.io.compress.brotli.BrotliCodec
|BZIP2|hbase.io.compress.bzip2.codec|org.apache.hadoop.io.compress.BZip2Codec
|GZ|hbase.io.compress.gz.codec|org.apache.hadoop.hbase.io.compress.ReusableStreamGzipCodec
|LZ4|hbase.io.compress.lz4.codec|org.apache.hadoop.io.compress.Lz4Codec
|LZMA|hbase.io.compress.lzma.codec|org.apache.hadoop.hbase.io.compress.xz.LzmaCodec
|LZO|hbase.io.compress.lzo.codec|com.hadoop.compression.lzo.LzoCodec
|SNAPPY|hbase.io.compress.snappy.codec|org.apache.hadoop.io.compress.SnappyCodec
|ZSTD|hbase.io.compress.zstd.codec|org.apache.hadoop.io.compress.ZStandardCodec
|===

The available codec implementation options are:

[options="header"]
|===
|Label|Codec implementation class|Notes
//----------------------
|BROTLI|org.apache.hadoop.hbase.io.compress.brotli.BrotliCodec|
Implemented with https://github.com/hyperxpro/Brotli4j[Brotli4j]
|BZIP2|org.apache.hadoop.io.compress.BZip2Codec|Hadoop native codec
|GZ|org.apache.hadoop.hbase.io.compress.ReusableStreamGzipCodec|
Requires the Hadoop native GZ codec
|LZ4|org.apache.hadoop.io.compress.Lz4Codec|Hadoop native codec
|LZ4|org.apache.hadoop.hbase.io.compress.aircompressor.Lz4Codec|
Pure Java implementation
|LZ4|org.apache.hadoop.hbase.io.compress.lz4.Lz4Codec|
Implemented with https://github.com/lz4/lz4-java[lz4-java]
|LZMA|org.apache.hadoop.hbase.io.compress.xz.LzmaCodec|
Implemented with https://tukaani.org/xz/java.html[XZ For Java]
|LZO|com.hadoop.compression.lzo.LzoCodec|Hadoop native codec,
requires GPL licensed native dependencies
|LZO|org.apache.hadoop.io.compress.LzoCodec|Hadoop native codec,
requires GPL licensed native dependencies
|LZO|org.apache.hadoop.hbase.io.compress.aircompressor.LzoCodec|
Pure Java implementation
|SNAPPY|org.apache.hadoop.io.compress.SnappyCodec|Hadoop native codec
|SNAPPY|org.apache.hadoop.hbase.io.compress.aircompressor.SnappyCodec|
Pure Java implementation
|SNAPPY|org.apache.hadoop.hbase.io.compress.xerial.SnappyCodec|
Implemented with https://github.com/xerial/snappy-java[snappy-java]
|ZSTD|org.apache.hadoop.io.compress.ZStandardCodec|Hadoop native codec
|ZSTD|org.apache.hadoop.hbase.io.compress.aircompressor.ZStdCodec|
Pure Java implementation, limited to a fixed compression level,
not data compatible with the Hadoop zstd codec
|ZSTD|org.apache.hadoop.hbase.io.compress.zstd.ZStdCodec|
Implemented with https://github.com/luben/zstd-jni[zstd-jni],
supports all compression levels, supports custom dictionaries
|===

Specify which codec implementation option you prefer for a given compression algorithm
in site configuration, like so:
[source]
----
...
<property>
<name>hbase.io.compress.lz4.codec</name>
<value>org.apache.hadoop.hbase.io.compress.lz4.Lz4Codec</value>
</property>
...
----

.Compressor Microbenchmarks

See https://github.com/apurtell/jmh-compression-tests

256MB (258,126,022 bytes exactly) of block data was extracted from two HFiles containing Common
Crawl data ingested using IntegrationLoadTestCommonCrawl, 2,680 blocks in total. This data was
processed by each new codec implementation as if the block data were being compressed again for
write into an HFile, but without writing any data, comparing only the CPU time and resource demand
of the codec itself. Absolute performance numbers will vary depending on hardware and software
particulars of your deployment. The relative differences are what are interesting. Measured time
is the average time in milliseconds required to compress all blocks of the 256MB file. This is
how long it would take to write the HFile containing these contents, minus the I/O overhead of
block encoding and actual persistence.

These are the results:

[options="header"]
|===
|Codec|Level|Time (milliseconds)|Result (bytes)|Improvement
//----------------------
|AirCompressor LZ4|-|349.989 ± 2.835|76,999,408|70.17%
|AirCompressor LZO|-|334.554 ± 3.243|79,369,805|69.25%
|AirCompressor Snappy|-|364.153 ± 19.718|80,201,763|68.93%
|AirCompressor Zstandard|3 (effective)|1108.267 ± 8.969|55,129,189|78.64%
|Brotli|1|593.107 ± 2.376|58,672,319|77.27%
|Brotli|3|1345.195 ± 27.327|53,917,438|79.11%
|Brotli|6|2812.411 ± 25.372|48,696,441|81.13%
|Brotli|10|74615.936 ± 224.854|44,970,710|82.58%
|LZ4 (lz4-java)|-|303.045 ± 0.783|76,974,364|70.18%
|LZMA|1|6410.428 ± 115.065|49,948,535|80.65%
|LZMA|3|8144.620 ± 152.119|49,109,363|80.97%
|LZMA|6|43802.576 ± 382.025|46,951,810|81.81%
|LZMA|9|49821.979 ± 580.110|46,951,810|81.81%
|Snappy (xerial)|-|360.225 ± 2.324|80,749,937|68.72%
|Zstd (zstd-jni)|1|654.699 ± 16.839|56,719,994|78.03%
|Zstd (zstd-jni)|3|839.160 ± 24.906|54,573,095|78.86%
|Zstd (zstd-jni)|5|1594.373 ± 22.384|52,025,485|79.84%
|Zstd (zstd-jni)|7|2308.705 ± 24.744|50,651,554|80.38%
|Zstd (zstd-jni)|9|3659.677 ± 58.018|50,208,425|80.55%
|Zstd (zstd-jni)|12|8705.294 ± 58.080|49,841,446|80.69%
|Zstd (zstd-jni)|15|19785.646 ± 278.080|48,499,508|81.21%
|Zstd (zstd-jni)|18|47702.097 ± 442.670|48,319,879|81.28%
|Zstd (zstd-jni)|22|97799.695 ± 1106.571|48,212,220|81.32%
|===

.Compressor Support On the Master

@@ -257,22 +433,29 @@ If native libraries are not available and Java's GZIP is used, `Got brand-new co
See <<brand.new.compressor,brand.new.compressor>>).

[[lzo.compression]]
.Install LZO Support
.Install Hadoop Native LZO Support

HBase cannot ship with LZO because of incompatibility between HBase, which uses an Apache Software License (ASL) and LZO, which uses a GPL license.
HBase cannot ship with the Hadoop native LZO codc because of incompatibility between HBase, which uses an Apache Software License (ASL) and LZO, which uses a GPL license.
See the link:https://github.com/twitter/hadoop-lzo/blob/master/README.md[Hadoop-LZO at Twitter] for information on configuring LZO support for HBase.

If you depend upon LZO compression, consider configuring your RegionServers to fail to start if LZO is not available.
If you depend upon LZO compression, consider using the pure Java and ASL licensed
AirCompressor LZO codec option instead of the Hadoop native default, or configure your
RegionServers to fail to start if native LZO support is not available.
See <<hbase.regionserver.codecs,hbase.regionserver.codecs>>.

[[lz4.compression]]
.Configure LZ4 Support

LZ4 support is bundled with Hadoop.
Make sure the hadoop shared library (libhadoop.so) is accessible when you start HBase.
After configuring your platform (see <<hadoop.native.lib,hadoop.native.lib>>), you can make a symbolic link from HBase to the native Hadoop libraries.
This assumes the two software installs are colocated.
For example, if my 'platform' is Linux-amd64-64:
.Configure Hadoop Native LZ4 Support

LZ4 support is bundled with Hadoop and is the default LZ4 codec implementation.
It is not required that you make use of the Hadoop LZ4 codec. Our LZ4 codec implemented
with lz4-java offers superior performance, and the AirCompressor LZ4 codec offers a
pure Java option for use where native support is not available.

That said, if you prefer the Hadoop option, make sure the hadoop shared library
(libhadoop.so) is accessible when you start HBase.
After configuring your platform (see <<hadoop.native.lib,hadoop.native.lib>>), you can
make a symbolic link from HBase to the native Hadoop libraries. This assumes the two
software installs are colocated. For example, if my 'platform' is Linux-amd64-64:
[source,bourne]
----
$ cd $HBASE_HOME
@@ -287,10 +470,15 @@ hbase(main):003:0> alter 'TestTable', {NAME => 'info', COMPRESSION => 'LZ4'}
----

[[snappy.compression.installation]]
.Install Snappy Support
.Install Hadoop native Snappy Support

Snappy support is bundled with Hadoop and is the default Snappy codec implementation.
It is not required that you make use of the Hadoop Snappy codec. Our Snappy codec
implemented with Xerial Snappy offers superior performance, and the AirCompressor
Snappy codec offers a pure Java option for use where native support is not available.

HBase does not ship with Snappy support because of licensing issues.
You can install Snappy binaries (for instance, by using +yum install snappy+ on CentOS) or build Snappy from source.
That said, if you prefer the Hadoop codec option, you can install Snappy binaries (for
instance, by using +yum install snappy+ on CentOS) or build Snappy from source.
After installing Snappy, search for the shared library, which will be called _libsnappy.so.X_ where X is a number.
If you built from source, copy the shared library to a known location on your system, such as _/opt/snappy/lib/_.